Nov 02 07:23:34.607607 ip-10-0-137-156 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 07:23:35.020670 ip-10-0-137-156 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:35.020670 ip-10-0-137-156 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 07:23:35.020670 ip-10-0-137-156 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:35.020670 ip-10-0-137-156 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026437    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026538    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026542    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026547    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026550    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026553    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.026544 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026556    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026559    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026561    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026564    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026567    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026570    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026572    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026576    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026603    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026607    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026611    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026614    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026617    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026620    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026624    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026627    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026629    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026632    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026635    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.026817 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026638    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026640    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026643    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026645    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026648    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026651    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026654    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026656    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026659    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026662    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026665    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026667    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026670    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026673    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026675    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026679    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026682    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026685    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026687    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026690    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.027262 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026693    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026696    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026699    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026702    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026705    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026716    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026719    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026722    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026725    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026727    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026730    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026734    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026738    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026741    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026744    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026747    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026750    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026752    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026755    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.027747 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026758    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026760    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026763    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026766    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026769    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026771    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026774    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026776    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026780    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026784    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026786    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026789    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026791    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026794    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026797    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026800    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026802    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026805    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026807    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026816    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.028201 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026819    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026821    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026824    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026827    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026829    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026832    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026835    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026837    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026840    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026843    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026845    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026849    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026852    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026854    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026857    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026860    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026862    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026865    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026867    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026870    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.028794 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026873    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.026876    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027741    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027749    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027753    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027756    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027759    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027762    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027765    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027768    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027771    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027774    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027802    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027806    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027809    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027811    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027814    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027817    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027820    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027822    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.029299 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027825    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027830    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027833    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027837    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027840    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027842    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027847    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027852    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027855    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027858    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027862    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027864    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027868    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027871    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027874    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027877    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027880    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027882    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027885    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.029786 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027888    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027890    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027893    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027895    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027898    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027901    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027909    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027913    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027917    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027920    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027923    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027925    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027928    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027931    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027934    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027936    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027939    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027942    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027945    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.030248 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027947    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027950    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027953    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027955    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027958    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027961    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027963    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027966    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027969    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027971    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027974    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027976    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027979    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027981    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027984    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027987    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027990    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027993    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027995    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.030720 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.027998    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028007    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028010    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028013    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028015    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028018    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028021    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028024    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028027    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028029    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028032    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028035    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028038    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028040    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028042    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028045    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028048    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028051    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028053    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028056    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.031168 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028058    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028061    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028064    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028067    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028069    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028072    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028074    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028077    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028080    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028082    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028085    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028637    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028644    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028648    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028651    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028654    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028657    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028660    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028664    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.031705 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028669    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028672    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028676    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028679    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028682    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028685    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028687    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028690    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028692    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028695    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028698    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028700    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028703    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028706    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028709    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028711    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028714    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028716    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028719    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.032183 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028722    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028724    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028727    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028730    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028733    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028736    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028738    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028741    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028745    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028750    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028753    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028756    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028759    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028761    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028764    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028767    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028769    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028772    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028775    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028778    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.032645 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028781    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028784    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028786    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028789    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028792    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028794    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028797    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028801    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028804    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028806    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028809    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028812    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028815    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028818    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028820    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028823    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028826    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028829    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028831    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.033129 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028834    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028836    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028842    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028845    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028847    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028850    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028853    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028856    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028858    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028861    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028864    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028866    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028869    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028871    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028874    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028876    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028879    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028882    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028884    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028887    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.033582 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028890    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028893    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028896    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028900    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028903    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028906    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028908    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028911    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028914    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028916    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028919    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028925    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028928    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028932    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028935    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028937    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028940    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028943    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.034094 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.028945    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029049    2573 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029058    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029064    2573 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029070    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029075    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029078    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029083    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029088    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029091    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029094    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029098    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029102    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029105    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029108    2573 flags.go:64] FLAG: --cgroup-root=""
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029111    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029114    2573 flags.go:64] FLAG: --client-ca-file=""
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029117    2573 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029120    2573 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029125    2573 flags.go:64] FLAG: --cluster-domain=""
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029129    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029132    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029135    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029140    2573 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 07:23:35.034568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029149    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029152    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029155    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029160    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029164    2573 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029168    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029171    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029174    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029177    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029182    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029186    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029189    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029192    2573 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029195    2573 flags.go:64] FLAG: --enable-server="true"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029198    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029203    2573 flags.go:64] FLAG: --event-burst="100"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029206    2573 flags.go:64] FLAG: --event-qps="50"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029209    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029213    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029216    2573 flags.go:64] FLAG: --eviction-hard=""
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029220    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029224    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029227    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029230    2573 flags.go:64] FLAG: --eviction-soft=""
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029233    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 07:23:35.035137 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029237    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029240    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029243    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029249    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029252    2573 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029255    2573 flags.go:64] FLAG: --feature-gates=""
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029259    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029263    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029266    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029269    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029274    2573 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029277    2573 flags.go:64] FLAG: --help="false"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029280    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029285    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029288    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029291    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029295    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029298    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029301    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029318    2573 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029323    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029328    2573 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029333    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029337    2573 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 07:23:35.035765 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029340    2573 flags.go:64] FLAG: --kube-reserved=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029343    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029346    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029349    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029352    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029356    2573 flags.go:64] FLAG: --lock-file=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029358    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029362    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029365    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029372    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029375    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029379    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029383    2573 flags.go:64] FLAG: --logging-format="text"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029386    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029389    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029393    2573 flags.go:64] FLAG: --manifest-url=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029396    2573 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029401    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029406    2573 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029410    2573 flags.go:64] FLAG: --max-pods="110"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029413    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029416    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029421    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029424    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029427    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 07:23:35.036433 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029430    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029433    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029441    2573 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029444    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029448    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029451    2573 flags.go:64] FLAG: --pod-cidr=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029454    2573 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029457    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029460    2573 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029463    2573 flags.go:64] FLAG: --port="10250"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029466    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029469    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01514a35ac383ddcf"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029472    2573 flags.go:64] FLAG: --qos-reserved=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029476    2573 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029479    2573 flags.go:64] FLAG: --register-node="true"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029482    2573 flags.go:64] FLAG: --register-with-taints=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029486    2573 flags.go:64] FLAG: --registry-burst="10"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029489    2573 flags.go:64] FLAG: --registry-qps="5"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029493    2573 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029496    2573 flags.go:64] FLAG: --reserved-memory=""
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029500    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029503    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029507    2573 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029510    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029513    2573 flags.go:64] FLAG: --runonce="false"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029517    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 07:23:35.037048 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029521    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029524    2573 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029527    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029530    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029534    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029538    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029541    2573 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029544    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029547    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029551    2573 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029554    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029557    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029560    2573 flags.go:64] FLAG: --system-cgroups=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029563    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029568    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029571    2573 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029575    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029579    2573 flags.go:64] FLAG: --tls-min-version=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029582    2573 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029585    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029588    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029591    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029595    2573 flags.go:64] FLAG: --v="2"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029600    2573 flags.go:64] FLAG: --version="false"
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029606    2573 flags.go:64] FLAG: --vmodule=""
Nov 02 07:23:35.037729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029610    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.029614    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029761    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029765    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029769    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029772    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029777    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029780    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029783    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029786    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029789    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029791    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029795    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029798    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029801    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029804    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029806    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029809    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029812    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029814    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.038336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029817    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029820    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029823    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029825    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029828    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029831    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029833    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029836    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029839    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029842    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029845    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029849    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029851    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029854    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029857    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029860    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029862    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029865    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029868    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029871    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.038816 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029874    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029877    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029880    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029882    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029885    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029888    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029891    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029894    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029896    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029899    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029902    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029904    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029907    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029910    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029912    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029915    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029918    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029920    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029923    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029925    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.039292 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029928    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029930    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029933    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029937    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029940    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029943    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029945    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029948    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029951    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029953    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029956    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029959    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029962    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029965    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029968    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029970    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029973    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029976    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029980    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.039788 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029983    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029985    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029988    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029991    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029994    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029996    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.029999    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030001    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030004    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030007    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030009    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030012    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030015    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030017    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030020    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030023    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030026    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030029    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030033    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.040232 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030038    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030041    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030045    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030048    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030052    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030055    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030058    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030062    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030065    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030068    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.030097    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030136    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030140    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030144    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030147    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030150    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.040769 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030152    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030155    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030158    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030160    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030164    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030169    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030171    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030175    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030179    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030182    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030185    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030188    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030190    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030193    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030197    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030201    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030204    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030206    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030209    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.041157 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030212    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030215    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030218    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030221    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030223    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030226    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030229    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030232    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030235    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030237    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030240    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030243    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030246    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030249    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030251    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030254    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030257    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030260    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030263    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.041626 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030266    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030268    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030271    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030274    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030276    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030279    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030282    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030284    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030288    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030291    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030293    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030296    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030298    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030301    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030319    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030323    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030326    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030328    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030331    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030334    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.042069 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030336    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030339    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030342    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030344    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030347    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030350    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030353    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030356    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030358    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030361    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030364    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030367    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030369    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030372    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030375    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030378    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030381    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030383    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030386    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030389    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.042570 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030391    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030394    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030397    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030399    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030402    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030405    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030409    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030412    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030415    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030417    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030420    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030423    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030425    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030428    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030431    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030433    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030436    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030439    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030443    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.043085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030445    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030448    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030451    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.030453    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.038094    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.038115    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.038131    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.038136    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038202    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038208    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038212    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038215    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038218    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038222    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038225    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038227    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038230    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038233    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038236    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038239    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.043648 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038242    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038244    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038247    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038250    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038253    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038255    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038258    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038261    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038264    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038266    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038269    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038272    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038274    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038277    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038280    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038283    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038288    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038298    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038302    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.044134 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038322    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038325    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038329    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038331    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038335    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038340    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038343    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038347    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038350    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038353    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038356    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038359    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038362    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038364    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038367    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038370    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038373    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038375    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038378    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.044591 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038380    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038383    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038386    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038389    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038391    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038394    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038397    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038400    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038402    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038405    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038409    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038411    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038415    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038418    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038421    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038423    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038426    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038429    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038432    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.045021 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038434    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038437    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038439    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038442    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038445    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038448    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038450    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038453    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038456    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038458    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038461    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038464    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038466    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038469    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038471    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038474    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038477    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038479    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038482    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038485    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.045491 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038487    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038490    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038492    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038497    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038500    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038503    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038506    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038509    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038511    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038514    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038517    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038519    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038522    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038524    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038527    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038530    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.045953 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038532    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.038564    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038723    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038728    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038731    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038733    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038737    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038739    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038742    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038745    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038748    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038750    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038753    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038756    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038759    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038761    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.046351 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038764    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038767    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038769    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038772    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038775    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038777    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038780    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038783    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038786    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038789    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038791    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038794    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038797    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038799    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038802    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038805    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038808    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038810    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038813    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038815    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.046727 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038818    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038821    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038824    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038827    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038830    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038833    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038836    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038839    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038842    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038844    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038847    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038850    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038853    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038855    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038858    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038861    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038864    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038866    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038869    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038871    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.047220 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038875    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038877    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038880    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038882    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038885    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038888    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038890    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038895    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038899    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038902    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038905    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038908    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038910    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038913    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038916    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038918    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038921    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038924    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038926    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.047702 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038929    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038931    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038935    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038938    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038944    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038947    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038950    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038952    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038955    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038958    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038960    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038963    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038966    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038968    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038971    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038974    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038977    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038980    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038983    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.048158 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038985    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038988    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038991    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038994    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038996    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.038999    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039002    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039004    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039007    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039010    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039012    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039016    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039020    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039023    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.039046    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039086    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:35.048674 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039091    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039094    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039098    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039101    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039104    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039106    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039109    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039112    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039114    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039117    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039119    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039122    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039126    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039129    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039132    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039134    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039137    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039140    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039142    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039145    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:35.049048 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039148    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039150    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039153    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039156    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039159    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039162    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039164    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039167    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039170    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039173    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039175    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039178    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039180    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039183    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039187    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039190    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039192    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039195    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039197    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039200    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:35.049555 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039203    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039206    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039208    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039211    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039214    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039217    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039221    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039226    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039229    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039232    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039236    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039239    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039241    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039244    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039246    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039249    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039252    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039254    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039257    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:35.050405 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039259    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039262    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039265    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039267    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039270    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039273    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039275    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039279    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039282    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039284    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039287    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039290    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039292    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039295    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039298    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039301    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039318    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039325    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039329    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:35.051126 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039332    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039335    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039339    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039341    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039344    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039347    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039350    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039352    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039356    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039358    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039362    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039366    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039369    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039372    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039375    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039377    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039380    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039383    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039387    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039389    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:35.051698 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039394    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039396    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039399    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039402    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039404    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039407    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:35.039410    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.039470    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.041985    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.043101    2573 server.go:1014] "Starting client certificate rotation"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.043199    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.043986    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.045410    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:23:35.052208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.049810    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:23:35.063118 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.063100    2573 log.go:25] "Validated CRI v1 runtime API"
Nov 02 07:23:35.067024 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.067011    2573 log.go:25] "Validated CRI v1 image API"
Nov 02 07:23:35.067882 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.067811    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 07:23:35.072648 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.072623    2573 fs.go:135] Filesystem UUIDs: map[0021f34d-aa3b-4fca-8f1d-ed3c2cc5f08f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c70c9398-4a38-48ad-9a50-3cb2f9a7fe2a:/dev/nvme0n1p4]
Nov 02 07:23:35.072730 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.072645    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 07:23:35.079361 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.079225    2573 manager.go:217] Machine: {Timestamp:2026-11-02 07:23:35.07774645 +0000 UTC m=+0.383508304 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099920 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21422a3dfcebed0dea53ef19b98ba7 SystemUUID:ec21422a-3dfc-ebed-0dea-53ef19b98ba7 BootID:024b276f-4117-4fb0-b70f-0d0c3f42df7a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:d3:be:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:d3:be:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:af:33:48:57:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 07:23:35.079361 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.079355    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 07:23:35.079472 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.079447    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 07:23:35.082918 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.082886    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 07:23:35.083136 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.082921    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 07:23:35.083136 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.083099    2573 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 07:23:35.083136 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.083110    2573 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 07:23:35.083280 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.083165    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:23:35.083280 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.083192    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:23:35.084537 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.084523    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 07:23:35.089407 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.089393    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 07:23:35.089561 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.089550    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 07:23:35.090441 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090430    2573 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 07:23:35.090494 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090450    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 07:23:35.090494 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090466    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 07:23:35.090494 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090480    2573 kubelet.go:405] "Adding apiserver pod source"
Nov 02 07:23:35.090494 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090492    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 07:23:35.090854 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.090840    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:23:35.093496 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.093470    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 07:23:35.100381 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.100363    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:23:35.101240 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101227    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 07:23:35.101282 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101259    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 07:23:35.101474 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101463    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 07:23:35.101510 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101480    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 07:23:35.101510 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101488    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 07:23:35.101510 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101497    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 07:23:35.101510 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101506    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 07:23:35.101611 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.101512    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 07:23:35.102160 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102152    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 07:23:35.102188 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102161    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 07:23:35.102188 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102169    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 07:23:35.102188 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102175    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 07:23:35.102188 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102184    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 07:23:35.102351 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102193    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 07:23:35.102914 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102902    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 07:23:35.102949 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.102915    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 07:23:35.108790 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.108764    2573 server.go:1263] "Started kubelet"
Nov 02 07:23:35.108887 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.108790    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 07:23:35.108887 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.108806    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 07:23:35.108887 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.108869    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 07:23:35.113931 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.112951    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 07:23:35.113357 ip-10-0-137-156 systemd[1]: Started Kubernetes Kubelet.
Nov 02 07:23:35.114536 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.114512    2573 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 07:23:35.118660 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.118633    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 07:23:35.123661 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.123641    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 07:23:35.125089 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125076    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 07:23:35.125262 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.125241    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.125884 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125868    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 07:23:35.125884 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125878    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 07:23:35.126014 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125892    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 07:23:35.126014 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125950    2573 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 07:23:35.126014 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125956    2573 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 07:23:35.126014 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125968    2573 factory.go:55] Registering systemd factory
Nov 02 07:23:35.126014 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.125991    2573 factory.go:223] Registration of the systemd container factory successfully
Nov 02 07:23:35.126227 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.126184    2573 factory.go:153] Registering CRI-O factory
Nov 02 07:23:35.126227 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.126195    2573 factory.go:223] Registration of the crio container factory successfully
Nov 02 07:23:35.126321 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.126238    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 07:23:35.126321 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.126255    2573 factory.go:103] Registering Raw factory
Nov 02 07:23:35.126321 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.126267    2573 manager.go:1199] Started watching for new ooms in manager
Nov 02 07:23:35.127334 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.127319    2573 manager.go:319] Starting recovery of all containers
Nov 02 07:23:35.141330 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.138359    2573 manager.go:324] Recovery completed
Nov 02 07:23:35.142721 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.142705    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.144445 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.144413    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.144519 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.144463    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.144519 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.144480    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.144989 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.144967    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 07:23:35.145140 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145124    2573 cpu_manager.go:226] "Starting" policy="none"
Nov 02 07:23:35.145140 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145137    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 07:23:35.145237 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145143    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.145237 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145158    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 07:23:35.145870 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145854    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.145947 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145883    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.145947 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.145897    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.146132 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.144969    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-156.ec2.internal.18e42987dd2fba04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-156.ec2.internal,},FirstTimestamp:2026-11-02 07:23:35.108516356 +0000 UTC m=+0.414278209,LastTimestamp:2026-11-02 07:23:35.108516356 +0000 UTC m=+0.414278209,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-156.ec2.internal,}"
Nov 02 07:23:35.147379 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.147365    2573 policy_none.go:50] "Start"
Nov 02 07:23:35.147379 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.147381    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 07:23:35.147460 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.147395    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 07:23:35.149446 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.149435    2573 policy_none.go:44] "Start"
Nov 02 07:23:35.153400 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.153246    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 07:23:35.153489 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.153346    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l5pbf"
Nov 02 07:23:35.157904 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.157877    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 07:23:35.158129 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.158112    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l5pbf"
Nov 02 07:23:35.166178 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.166151    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 07:23:35.166178 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.166089    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-156.ec2.internal.18e42987df53f311  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-156.ec2.internal,},FirstTimestamp:2026-11-02 07:23:35.144444689 +0000 UTC m=+0.450206549,LastTimestamp:2026-11-02 07:23:35.144444689 +0000 UTC m=+0.450206549,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-156.ec2.internal,}"
Nov 02 07:23:35.166384 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.166318    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 07:23:35.166384 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.166358    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 07:23:35.184111 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.184074    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 07:23:35.189674 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.189662    2573 manager.go:341] "Starting Device Plugin manager"
Nov 02 07:23:35.189734 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.189690    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 07:23:35.189734 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.189699    2573 server.go:87] "Starting device plugin registration server"
Nov 02 07:23:35.189974 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.189962    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 07:23:35.190011 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.189975    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 07:23:35.190106 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.190080    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 07:23:35.190242 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.190167    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 07:23:35.190242 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.190175    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 07:23:35.190898 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.190877    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 07:23:35.190962 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.190921    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.257159 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.257126    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 07:23:35.259183 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.258367    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 07:23:35.259183 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.258396    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 07:23:35.259183 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.258422    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 07:23:35.259183 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.258467    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 07:23:35.272534 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.272472    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:35.290369 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.290341    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.291339 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.291303    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.291443 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.291355    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.291443 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.291369    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.291443 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.291393    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.308182 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.308160    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.308303 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.308187    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-156.ec2.internal\": node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.349648 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.349625    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.358539 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.358509    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"]
Nov 02 07:23:35.358622 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.358573    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.359459 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.359444    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.359550 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.359471    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.359550 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.359484    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.361645 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.361633    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.361807 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.361795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.361852 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.361824    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.362342 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362321    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.362428 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362353    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.362428 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362364    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.362428 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362380    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.362428 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362402    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.362428 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.362414    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.364370 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.364355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.364460 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.364380    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:35.365110 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.365093    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:35.365166 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.365125    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:35.365166 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.365141    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:35.397862 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.397847    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-156.ec2.internal\" not found" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.400978 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.400960    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-156.ec2.internal\" not found" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.427640 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.427615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.427725 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.427647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.427725 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.427675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.449697 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.449674    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.528426 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.528426 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.528426 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.528609 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528457    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.528609 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528467    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.528609 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.528473    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.549810 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.549778    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.650893 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.650850    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.701071 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.701046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.703977 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:35.703958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:35.751524 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.751489    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.852120 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.852080    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:35.954073 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:35.954016    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:36.042467 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.042369    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 07:23:36.042961 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.042542    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 07:23:36.055201 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:36.055171    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:23:36.088440 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.088407    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:36.092739 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.092719    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:36.114686 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.114660    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:36.124366 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.124332    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 07:23:36.125520 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.125502    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:36.147660 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.147633    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:23:36.148430 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.148407    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:23:36.163665 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.163625    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:35 +0000 UTC" deadline="2028-05-25 00:28:22.210236308 +0000 UTC"
Nov 02 07:23:36.163665 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.163660    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13673h4m46.046579774s"
Nov 02 07:23:36.165936 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.165925    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:23:36.208711 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.208674    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 07:23:36.208866 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.208742    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lnmcq"
Nov 02 07:23:36.221693 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.221662    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lnmcq"
Nov 02 07:23:36.246514 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:36.246468    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb271a57358807e5a4d178ecbda3280fd.slice/crio-79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3 WatchSource:0}: Error finding container 79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3: Status 404 returned error can't find the container with id 79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3
Nov 02 07:23:36.256146 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.256127    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:23:36.262091 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.262051    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerStarted","Data":"79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3"}
Nov 02 07:23:36.270188 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:36.270162    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735495427ccc6de5684bdc4a015d248e.slice/crio-e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17 WatchSource:0}: Error finding container e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17: Status 404 returned error can't find the container with id e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17
Nov 02 07:23:36.872589 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:36.872557    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:37.092754 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.092727    2573 apiserver.go:52] "Watching apiserver"
Nov 02 07:23:37.129729 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.129647    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 07:23:37.130064 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.130038    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-97v78","openshift-image-registry/node-ca-mldbl","openshift-multus/multus-wxks2","kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l","openshift-network-operator/iptables-alerter-vnz9b","openshift-ovn-kubernetes/ovnkube-node-cljfx","openshift-cluster-node-tuning-operator/tuned-cwm7p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal","openshift-multus/network-metrics-daemon-qnkkt","kube-system/global-pull-secret-syncer-brmn4","openshift-multus/multus-additional-cni-plugins-6bt4h","openshift-network-diagnostics/network-check-target-xq8m6","kube-system/konnectivity-agent-qfghm"]
Nov 02 07:23:37.132953 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.132919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.133105 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.133012    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:37.135415 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.135270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:37.135415 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.135366    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:37.137686 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.137658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.139954 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.139930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.142350 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.142319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.145072 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.144701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.148795 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.148189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.149843 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.149819    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.151506 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.151483    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.151633 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.151517    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 07:23:37.151703 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.151638    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 07:23:37.151703 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.151664    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.151837 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.151729    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\""
Nov 02 07:23:37.152496 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.152226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.154625 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.154476    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 07:23:37.154625 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.154493    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 07:23:37.154625 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.154595    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\""
Nov 02 07:23:37.155110 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.155043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.155542 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.155492    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 07:23:37.155656 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.155562    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.155737 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.155685    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.155781 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.155767    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\""
Nov 02 07:23:37.157373 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.157354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.157481 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.157439    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:37.158067 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.158045    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.158156 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.158134    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.158263 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.158240    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 07:23:37.158378 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.158273    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\""
Nov 02 07:23:37.160073 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.160051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.161968 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.161947    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.161968 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.161958    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.162124 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.161981    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\""
Nov 02 07:23:37.162931 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.162848    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 07:23:37.163095 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.163033    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.163095 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.163069    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 07:23:37.164510 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.164483    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 07:23:37.226031 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.225966    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:36 +0000 UTC" deadline="2028-07-14 05:47:04.711089924 +0000 UTC"
Nov 02 07:23:37.226031 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.226025    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14878h23m27.485069166s"
Nov 02 07:23:37.227172 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.227153    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 07:23:37.237077 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237049    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237203 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237082    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237203 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7zj\" (UniqueName: \"kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237203 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.237404 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.237404 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.237404 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.237404 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237335    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.237579 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lbm9\" (UniqueName: \"kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.237761 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237823    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.237963 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.237991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghdw\" (UniqueName: \"kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsnh5\" (UniqueName: \"kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.238157 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238211    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgsr\" (UniqueName: \"kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.238458 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6js\" (UniqueName: \"kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85d7r\" (UniqueName: \"kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.238963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239032    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239127    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6km\" (UniqueName: \"kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9cw\" (UniqueName: \"kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239347    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.239600 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.240461 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.239497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.260142 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260105    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 07:23:37.260142 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260128    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 07:23:37.260142 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260140    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260113    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260132    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260110    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260106    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260267    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\""
Nov 02 07:23:37.260419 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260383    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\""
Nov 02 07:23:37.260650 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260444    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.260650 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260518    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\""
Nov 02 07:23:37.260650 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260602    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 07:23:37.260783 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.260686    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\""
Nov 02 07:23:37.264719 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.264690    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal" event={"ID":"735495427ccc6de5684bdc4a015d248e","Type":"ContainerStarted","Data":"e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17"}
Nov 02 07:23:37.340625 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.340625 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340807    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbm9\" (UniqueName: \"kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.340878 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.340975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341022    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghdw\" (UniqueName: \"kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsnh5\" (UniqueName: \"kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.341391 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgsr\" (UniqueName: \"kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341526    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6js\" (UniqueName: \"kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.342085 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85d7r\" (UniqueName: \"kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.341986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6km\" (UniqueName: \"kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.342785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342113    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9cw\" (UniqueName: \"kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342449    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342472    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342474    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7zj\" (UniqueName: \"kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342567    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.343653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342613    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342618    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342730    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342735    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342766    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342784    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342798    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342825    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342833    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342831    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342847    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.342857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.342994    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343007    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343053    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.343094    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:23:37.843061607 +0000 UTC m=+3.148823447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343155    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.344581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343544    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343623    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343780    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.343974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344048    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344084    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344452    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344471    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344513    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344586    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.344968    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345022    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345045    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345120    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.345545 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345220    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345731    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345793    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345790    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345835    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345870    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345911    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345921    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.345961    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.345998    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346006    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.346050    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:37.846034173 +0000 UTC m=+3.151796046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346140    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.346288 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346170    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346784    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346848    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346881    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346888    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.346936    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.347112 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347010    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347183    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347232    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347297    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347345    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347427    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347474    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347524    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347497    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347566    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347620    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347667    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347721    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.348416 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.347850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.349843 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.349818    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.349992 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.349969    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.350261 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.350239    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.353532 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.353485    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.367613 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.367579    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:37.367613 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.367618    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:37.367814 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.367635    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:37.367814 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.367706    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:37.867684464 +0000 UTC m=+3.173446330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:37.369404 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.369366    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6km\" (UniqueName: \"kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.369730 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.369706    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85d7r\" (UniqueName: \"kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.370903 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.370875    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbm9\" (UniqueName: \"kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.371632 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.371585    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7zj\" (UniqueName: \"kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.371876 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.371857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgsr\" (UniqueName: \"kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.373081 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.373058    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsnh5\" (UniqueName: \"kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.373204 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.373143    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6js\" (UniqueName: \"kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.373204 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.373187    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9cw\" (UniqueName: \"kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.375736 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.375683    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghdw\" (UniqueName: \"kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.454171 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.454093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wxks2"
Nov 02 07:23:37.465085 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.465052    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec1abc9_0c41_4337_b497_b4ddff25663d.slice/crio-3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f WatchSource:0}: Error finding container 3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f: Status 404 returned error can't find the container with id 3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f
Nov 02 07:23:37.467905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.467881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:23:37.476560 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.476528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:37.480275 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.480244    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a44cf5_6890_4936_861f_7f26de059789.slice/crio-1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041 WatchSource:0}: Error finding container 1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041: Status 404 returned error can't find the container with id 1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041
Nov 02 07:23:37.482489 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.482467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:23:37.490463 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.490442    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:23:37.500755 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.498921    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f836f8b_5b38_4697_b577_46cc16c02f12.slice/crio-8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07 WatchSource:0}: Error finding container 8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07: Status 404 returned error can't find the container with id 8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07
Nov 02 07:23:37.500755 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.499835    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2780dc0b_f641_45c0_86f8_769a72bf589c.slice/crio-9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566 WatchSource:0}: Error finding container 9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566: Status 404 returned error can't find the container with id 9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566
Nov 02 07:23:37.503189 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.503094    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb95b14_6d7c_4241_b22a_dd67fa387dd3.slice/crio-251d92649b54ef38e24d7eecee6543a6b6944edde9e87504585a23b6fabe51e4 WatchSource:0}: Error finding container 251d92649b54ef38e24d7eecee6543a6b6944edde9e87504585a23b6fabe51e4: Status 404 returned error can't find the container with id 251d92649b54ef38e24d7eecee6543a6b6944edde9e87504585a23b6fabe51e4
Nov 02 07:23:37.512899 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.512873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:23:37.515341 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.515303    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97v78"
Nov 02 07:23:37.519519 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.519453    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:37.534417 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.534056    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b02e54_cab8_41d7_909c_7f0cbe5eab71.slice/crio-c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc WatchSource:0}: Error finding container c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc: Status 404 returned error can't find the container with id c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc
Nov 02 07:23:37.534529 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:23:37.534468    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d98a9be_246b_45a0_b64c_71119a497445.slice/crio-d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830 WatchSource:0}: Error finding container d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830: Status 404 returned error can't find the container with id d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830
Nov 02 07:23:37.536341 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.536180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:23:37.851518 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.851481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:37.851628 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.851537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:37.851669 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.851642    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:37.851669 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.851656    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:37.851735 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.851710    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:23:38.851694002 +0000 UTC m=+4.157455858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:37.851735 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.851727    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:38.851718994 +0000 UTC m=+4.157480835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:37.953656 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:37.952937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:37.953656 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.953152    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:37.953656 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.953174    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:37.953656 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.953188    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:37.953656 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:37.953257    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:38.953236277 +0000 UTC m=+4.258998133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:38.226989 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.226601    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:36 +0000 UTC" deadline="2028-04-15 07:33:36.565377405 +0000 UTC"
Nov 02 07:23:38.226989 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.226921    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12720h9m58.338463713s"
Nov 02 07:23:38.260568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.259656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:38.260568 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.259814    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:38.260568 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.260269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:38.260568 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.260473    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:38.289883 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.289688    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830"}
Nov 02 07:23:38.300566 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.300300    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97v78" event={"ID":"71b02e54-cab8-41d7-909c-7f0cbe5eab71","Type":"ContainerStarted","Data":"c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc"}
Nov 02 07:23:38.304303 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.303993    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mldbl" event={"ID":"903d010c-458d-4c3a-8c94-8bf685cb6ef3","Type":"ContainerStarted","Data":"f4468fdc21b1dacf7baf50dcc9c5f6e811fc39441acada88892cfb7703dc006f"}
Nov 02 07:23:38.308857 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.308771    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnz9b" event={"ID":"2780dc0b-f641-45c0-86f8-769a72bf589c","Type":"ContainerStarted","Data":"9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566"}
Nov 02 07:23:38.317858 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.317821    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal" event={"ID":"735495427ccc6de5684bdc4a015d248e","Type":"ContainerStarted","Data":"bf04a3b56358c5ef4f2d8c9d90084db2d0148080d4dab07a51eadd6dec0401a9"}
Nov 02 07:23:38.329540 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.329500    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerStarted","Data":"763de3ed6b90822725f6ce67d1fd4b84ae7caae15dfaef9c67f6d0716423c94e"}
Nov 02 07:23:38.335673 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.335644    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwm7p" event={"ID":"7cb95b14-6d7c-4241-b22a-dd67fa387dd3","Type":"ContainerStarted","Data":"251d92649b54ef38e24d7eecee6543a6b6944edde9e87504585a23b6fabe51e4"}
Nov 02 07:23:38.345739 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.345716    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qfghm" event={"ID":"0f836f8b-5b38-4697-b577-46cc16c02f12","Type":"ContainerStarted","Data":"8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07"}
Nov 02 07:23:38.347429 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.347401    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l" event={"ID":"43a44cf5-6890-4936-861f-7f26de059789","Type":"ContainerStarted","Data":"1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041"}
Nov 02 07:23:38.351563 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.351533    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wxks2" event={"ID":"3ec1abc9-0c41-4337-b497-b4ddff25663d","Type":"ContainerStarted","Data":"3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f"}
Nov 02 07:23:38.860164 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.860058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:38.860378 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.860166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:38.860378 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.860309    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:38.860504 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.860393    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:23:40.860373603 +0000 UTC m=+6.166135459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:38.860970 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.860843    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:38.860970 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.860907    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:40.86088849 +0000 UTC m=+6.166650340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:38.960974 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:38.960930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:38.961170 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.961144    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:38.961234 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.961183    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:38.961234 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.961198    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:38.961377 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:38.961272    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:40.961251408 +0000 UTC m=+6.267013269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:39.261351 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:39.261268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:39.261823 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:39.261424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:39.365061 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:39.365024    2573 generic.go:356] "Generic (PLEG): container finished" podID="b271a57358807e5a4d178ecbda3280fd" containerID="1a5a0a5fb24d88372ba4e841447d61150b771da5001791c42581b8bb58c38668" exitCode=0
Nov 02 07:23:39.366058 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:39.366025    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerDied","Data":"1a5a0a5fb24d88372ba4e841447d61150b771da5001791c42581b8bb58c38668"}
Nov 02 07:23:39.404336 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:39.404279    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal" podStartSLOduration=3.404257062 podStartE2EDuration="3.404257062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:38.356302085 +0000 UTC m=+3.662063954" watchObservedRunningTime="2026-11-02 07:23:39.404257062 +0000 UTC m=+4.710018928"
Nov 02 07:23:40.260019 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.259405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:40.260019 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.259542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:40.260019 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.259887    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:40.260019 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.259975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:40.373133 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.372463    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerStarted","Data":"65e30be68534007df3d5bada78ebb00ba15f4806ead9edcc207b74bc58c0803e"}
Nov 02 07:23:40.408685 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.408621    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" podStartSLOduration=4.408599482 podStartE2EDuration="4.408599482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:40.40806134 +0000 UTC m=+5.713823206" watchObservedRunningTime="2026-11-02 07:23:40.408599482 +0000 UTC m=+5.714361345"
Nov 02 07:23:40.877131 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.877025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:40.877131 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.877100    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:40.877388 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.877253    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:40.877388 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.877328    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:44.87729634 +0000 UTC m=+10.183058193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:40.877510 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.877419    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:40.877510 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.877465    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:23:44.877451369 +0000 UTC m=+10.183213223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:40.978485 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:40.978438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:40.978670 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.978640    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:40.978670 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.978669    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:40.978781 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.978683    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:40.978781 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:40.978746    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:44.978727913 +0000 UTC m=+10.284489771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:41.261074 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:41.260480    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:41.261074 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:41.260608    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:41.388023 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:41.387976    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="a61a66be103c06b14343b035b2b65f76f6d163b3c7f1ebbe3a5e813c3d783d72" exitCode=0
Nov 02 07:23:41.389016 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:41.388990    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"a61a66be103c06b14343b035b2b65f76f6d163b3c7f1ebbe3a5e813c3d783d72"}
Nov 02 07:23:42.259152 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:42.259117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:42.259349 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:42.259228    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:42.259485 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:42.259468    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:42.259555 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:42.259538    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:42.397024 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:42.396987    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qfghm" event={"ID":"0f836f8b-5b38-4697-b577-46cc16c02f12","Type":"ContainerStarted","Data":"b95e0534cc147c1e25ce1b3b47369a62af9cd64f53540c0c0b38f48f7261da26"}
Nov 02 07:23:43.260169 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:43.259630    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:43.260169 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:43.259778    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:43.401851 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:43.401813    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l" event={"ID":"43a44cf5-6890-4936-861f-7f26de059789","Type":"ContainerStarted","Data":"4536402982a55413744584171b9ec683b69fd44f0c47c66173349087e5b823a1"}
Nov 02 07:23:43.406129 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:43.406090    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mldbl" event={"ID":"903d010c-458d-4c3a-8c94-8bf685cb6ef3","Type":"ContainerStarted","Data":"5f5352f87e51b8d43d8f784d58f3e7d912a29e1120bbd56e0d1a130da272af97"}
Nov 02 07:23:43.441469 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:43.441409    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qfghm" podStartSLOduration=4.377207945 podStartE2EDuration="8.441388443s" totalImagesPullingTime="4.064180498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:42.433339568 +0000 UTC m=+7.739101429" watchObservedRunningTime="2026-11-02 07:23:43.441388443 +0000 UTC m=+8.747150306"
Nov 02 07:23:44.259383 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:44.259300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:44.259571 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.259461    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:44.260738 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:44.260708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:44.260984 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.260949    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:44.792802 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:44.792722    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:44.909903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:44.909976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.910100    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.910163    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:52.910142821 +0000 UTC m=+18.215904677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.910571    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:44.910733 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:44.910624    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:23:52.910608454 +0000 UTC m=+18.216370307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:45.010794 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.010761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:45.010977 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:45.010955    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:45.011060 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:45.010987    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:45.011060 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:45.011000    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:45.011159 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:45.011060    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:53.011041466 +0000 UTC m=+18.316803316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:45.202186 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.202043    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:23:44.792739085Z","UUID":"ef45ba70-13ba-422a-b433-5affeb4660cb","Handler":null,"Name":"","Endpoint":""}
Nov 02 07:23:45.204864 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.204355    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 07:23:45.204864 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.204382    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 07:23:45.262653 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.262625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:45.262809 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:45.262731    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:45.416155 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.416117    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l" event={"ID":"43a44cf5-6890-4936-861f-7f26de059789","Type":"ContainerStarted","Data":"4ef4b30959be6019de576f4e72c4ebbe1dfe786989bd3c537b209261cbeaddf9"}
Nov 02 07:23:45.417984 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.417948    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wxks2" event={"ID":"3ec1abc9-0c41-4337-b497-b4ddff25663d","Type":"ContainerStarted","Data":"f39323f6a57632346b9cc79c6308c4f8e74f83c25a326f60cf52784606522af2"}
Nov 02 07:23:45.456747 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:45.456667    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mldbl" podStartSLOduration=5.382395822 podStartE2EDuration="10.456649797s" totalImagesPullingTime="5.074253975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:43.441818626 +0000 UTC m=+8.747580488" watchObservedRunningTime="2026-11-02 07:23:45.456649797 +0000 UTC m=+10.762411660"
Nov 02 07:23:46.229952 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.229734    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:46.230490 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.230465    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:46.259425 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.259394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:46.259586 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:46.259528    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:46.259586 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.259579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:46.259716 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:46.259692    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:46.266009 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.265968    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wxks2" podStartSLOduration=3.722274816 podStartE2EDuration="11.265954654s" totalImagesPullingTime="7.543679838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:45.462549623 +0000 UTC m=+10.768311486" watchObservedRunningTime="2026-11-02 07:23:46.265954654 +0000 UTC m=+11.571716561"
Nov 02 07:23:46.423053 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.422989    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="f66ba86d5d6a30be75a53200f63c0525c2c39906edd11e1b702b371f1d6f2fa6" exitCode=0
Nov 02 07:23:46.423193 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.423085    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"f66ba86d5d6a30be75a53200f63c0525c2c39906edd11e1b702b371f1d6f2fa6"}
Nov 02 07:23:46.425856 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.425829    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l" event={"ID":"43a44cf5-6890-4936-861f-7f26de059789","Type":"ContainerStarted","Data":"c1f7cd2b3ab7041c00b130ce9fc8837a6d101e062799b6a0f22c0b2170c70a43"}
Nov 02 07:23:46.428973 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.428948    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97v78" event={"ID":"71b02e54-cab8-41d7-909c-7f0cbe5eab71","Type":"ContainerStarted","Data":"419af6ad8c4bdbff728181d18da29e10e3ec567909c747835703737b47060154"}
Nov 02 07:23:46.502352 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.502277    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l" podStartSLOduration=3.481908195 podStartE2EDuration="11.502258892s" totalImagesPullingTime="8.020350697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:46.501851932 +0000 UTC m=+11.807613794" watchObservedRunningTime="2026-11-02 07:23:46.502258892 +0000 UTC m=+11.808020752"
Nov 02 07:23:46.536785 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:46.536726    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-97v78" podStartSLOduration=3.634771095 podStartE2EDuration="11.53671066s" totalImagesPullingTime="7.901939565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:46.536337183 +0000 UTC m=+11.842099041" watchObservedRunningTime="2026-11-02 07:23:46.53671066 +0000 UTC m=+11.842472524"
Nov 02 07:23:47.258666 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.258647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:47.259074 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:47.258745    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:47.432220 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.432180    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnz9b" event={"ID":"2780dc0b-f641-45c0-86f8-769a72bf589c","Type":"ContainerStarted","Data":"7b298c3a0093dbfcba4fe64234b54ab3b7345e86b58c08662753059494a3d312"}
Nov 02 07:23:47.434712 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.434659    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="661d28dd7c6e9409f0b2e19866c2aaa158a2ed1e6001716206d8c7c25c843073" exitCode=0
Nov 02 07:23:47.434820 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.434736    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"661d28dd7c6e9409f0b2e19866c2aaa158a2ed1e6001716206d8c7c25c843073"}
Nov 02 07:23:47.436651 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.436541    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwm7p" event={"ID":"7cb95b14-6d7c-4241-b22a-dd67fa387dd3","Type":"ContainerStarted","Data":"b14b71c2d1803b702a6a0e4dd87140594dcb827303fc207b0cbfa66ab8cc3a38"}
Nov 02 07:23:47.466984 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.466928    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vnz9b" podStartSLOduration=4.371704 podStartE2EDuration="12.466910205s" totalImagesPullingTime="8.095206205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:47.466762697 +0000 UTC m=+12.772524557" watchObservedRunningTime="2026-11-02 07:23:47.466910205 +0000 UTC m=+12.772672069"
Nov 02 07:23:47.506896 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:47.506846    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cwm7p" podStartSLOduration=2.75773054 podStartE2EDuration="12.506831684s" totalImagesPullingTime="9.749101144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:47.506302478 +0000 UTC m=+12.812064338" watchObservedRunningTime="2026-11-02 07:23:47.506831684 +0000 UTC m=+12.812593547"
Nov 02 07:23:48.259066 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:48.259030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:48.259719 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:48.259030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:48.259719 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:48.259168    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:48.259719 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:48.259237    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:48.441721 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:48.441638    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="da2cc743d7a0e57dd6b905707a70218a8f3d3aa303e7d112ff4884f44bacbf0f" exitCode=0
Nov 02 07:23:48.441856 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:48.441718    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"da2cc743d7a0e57dd6b905707a70218a8f3d3aa303e7d112ff4884f44bacbf0f"}
Nov 02 07:23:49.259222 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:49.259186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:49.259793 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:49.259331    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:50.258751 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:50.258593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:50.258871 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:50.258637    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:50.258871 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:50.258845    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:50.258980 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:50.258930    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:50.450006 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:50.449969    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="1026a7ef615fddfeb0668d4810069708937c79be96738f9b29703345bc813eaa" exitCode=0
Nov 02 07:23:50.450524 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:50.450024    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"1026a7ef615fddfeb0668d4810069708937c79be96738f9b29703345bc813eaa"}
Nov 02 07:23:51.259111 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:51.259074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:51.259338 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:51.259202    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:51.456504 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:51.456467    2573 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="f4367a2bbbc46bbb6b861a5977ee4c322fc819828753f2e9d3a227c5ef23dbfc" exitCode=0
Nov 02 07:23:51.456917 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:51.456533    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerDied","Data":"f4367a2bbbc46bbb6b861a5977ee4c322fc819828753f2e9d3a227c5ef23dbfc"}
Nov 02 07:23:52.259606 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.259569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:52.259606 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.259600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:52.259963 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.259676    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:52.259963 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.259799    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:52.462518 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.462493    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" event={"ID":"27a575b2-ee3c-4647-b1a6-8b29faed0a41","Type":"ContainerStarted","Data":"05f458999607c13a59fd00673329661561a7c1f9d225c799a56361b9ff818bb6"}
Nov 02 07:23:52.525261 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.525217    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6bt4h" podStartSLOduration=7.537388454 podStartE2EDuration="17.525202743s" totalImagesPullingTime="9.987814289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:52.514638101 +0000 UTC m=+17.820399961" watchObservedRunningTime="2026-11-02 07:23:52.525202743 +0000 UTC m=+17.830964604"
Nov 02 07:23:52.972747 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.972509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:52.972747 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:52.972714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:52.972935 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.972663    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:52.972935 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.972816    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:52.972935 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.972826    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret podName:a3daa4be-adf6-4e9f-8307-98e3c0d180ec nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.972810532 +0000 UTC m=+34.278572393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret") pod "global-pull-secret-syncer-brmn4" (UID: "a3daa4be-adf6-4e9f-8307-98e3c0d180ec") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:52.972935 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:52.972855    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.972843809 +0000 UTC m=+34.278605648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:53.073276 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.073241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:53.073445 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:53.073376    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:53.073445 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:53.073390    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:53.073445 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:53.073400    2573 projected.go:196] Error preparing data for projected volume kube-api-access-4dlg9 for pod openshift-network-diagnostics/network-check-target-xq8m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:53.073445 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:53.073446    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9 podName:dc523525-bb72-4547-a2dc-c77957b278b0 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:09.073433635 +0000 UTC m=+34.379195474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dlg9" (UniqueName: "kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9") pod "network-check-target-xq8m6" (UID: "dc523525-bb72-4547-a2dc-c77957b278b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:53.259053 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.259021    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:53.259205 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:53.259147    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:53.467106 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467069    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"a9997eafd28998bf0fea44350675245ce1afa21a4197e36945340ac1c623e6e8"}
Nov 02 07:23:53.467508 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467141    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"8789970b81080e3a5919fa310063d9b4fe9bc7235310620452aa74966bc07137"}
Nov 02 07:23:53.467508 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467177    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"d356317fecb0725cb121a1ad7127fb8413dbd86e93a5a45b42c3b301f33a77e4"}
Nov 02 07:23:53.467508 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467206    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"0e8f6bc0f47725ce2a3474bdcd964726371865582bc19b13043fddce975b451d"}
Nov 02 07:23:53.467508 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467290    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"cbd4ba5a88898f8253442f5ef80e0d165128682084be855358b422ccdedcb59b"}
Nov 02 07:23:53.467508 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:53.467333    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"13642a9c8c5bbef9a768597867ebf7d47e3dbd348783e4008ece3dc44365dc2d"}
Nov 02 07:23:54.258935 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:54.258901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:54.258935 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:54.258932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:54.259109 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:54.259006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:54.259141 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:54.259114    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:55.260094 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:55.260071    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:55.260537 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:55.260148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:55.474403 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:55.474369    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"28d1945a5708bda6468e166e456ef4299e0b947acee1da3be3386131ecaa8559"}
Nov 02 07:23:56.259445 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:56.259418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:56.259597 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:56.259517    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:56.259597 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:56.259543    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:56.259597 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:56.259589    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:57.259297 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:57.259272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:57.259866 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:57.259395    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:23:57.482044 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:57.481999    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" event={"ID":"4d98a9be-246b-45a0-b64c-71119a497445","Type":"ContainerStarted","Data":"49bf47b8ab9198ca18912e0bfdf5c796c06d411de1c41998dda5106ecba92eb5"}
Nov 02 07:23:57.482355 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:57.482337    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:57.497578 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:57.497552    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:57.560670 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:57.560574    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx" podStartSLOduration=7.655839718 podStartE2EDuration="22.560562296s" totalImagesPullingTime="14.904722578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:57.560382274 +0000 UTC m=+22.866144135" watchObservedRunningTime="2026-11-02 07:23:57.560562296 +0000 UTC m=+22.866324156"
Nov 02 07:23:58.168136 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.168103    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:58.168324 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.168257    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 07:23:58.168862 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.168830    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:23:58.259517 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.259479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:58.260266 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.259530    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:58.260266 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:58.259655    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:58.260266 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:58.259825    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:58.484287 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.484205    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:58.484599 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.484583    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:58.499242 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:58.499212    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:23:59.090155 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.089891    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brmn4"]
Nov 02 07:23:59.090331 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.090213    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:23:59.090388 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:59.090351    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:23:59.090675 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.090516    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnkkt"]
Nov 02 07:23:59.090675 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.090625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:23:59.090792 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:59.090744    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:23:59.091134 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.091112    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq8m6"]
Nov 02 07:23:59.091232 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:23:59.091212    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:23:59.091342 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:23:59.091302    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:24:00.258908 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:00.258878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:00.259350 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:00.258989    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:24:01.259096 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:01.259070    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:01.259541 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:01.259078    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:01.259541 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:01.259207    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:24:01.259541 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:01.259276    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:24:02.259072 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:02.259043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:02.259270 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:02.259141    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:24:03.258870 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:03.258839    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:03.259237 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:03.258869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:03.259237 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:03.258958    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnkkt" podUID="d7a903fa-2c1d-47ed-8701-1c220c920b57"
Nov 02 07:24:03.259237 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:03.259040    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brmn4" podUID="a3daa4be-adf6-4e9f-8307-98e3c0d180ec"
Nov 02 07:24:04.258573 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.258534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:04.258750 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.258638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq8m6" podUID="dc523525-bb72-4547-a2dc-c77957b278b0"
Nov 02 07:24:04.489040 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.489014    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeReady"
Nov 02 07:24:04.489431 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.489141    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 07:24:04.545865 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.545789    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9dfxs"]
Nov 02 07:24:04.548674 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.548656    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2rzs7"]
Nov 02 07:24:04.548789 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.548772    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.551696 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.551676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.561327 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.561286    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 07:24:04.561457 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.561399    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 07:24:04.561457 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.561417    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 07:24:04.561531 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.561493    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\""
Nov 02 07:24:04.564413 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.564395    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 07:24:04.564413 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.564403    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 07:24:04.564536 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.564429    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\""
Nov 02 07:24:04.573494 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.573440    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dfxs"]
Nov 02 07:24:04.576132 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.576104    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rzs7"]
Nov 02 07:24:04.655930 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.655893    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh2r\" (UniqueName: \"kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.656103 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.655948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.656103 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.655973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.656180 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.656126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.656180 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.656154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.656250 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.656181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndwz\" (UniqueName: \"kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.751619 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.751582    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kc2t9"]
Nov 02 07:24:04.756845 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.756817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.756845 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.756849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.756999 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.756872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.757033 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.756999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.757074 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.757046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndwz\" (UniqueName: \"kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.757141 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.757125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh2r\" (UniqueName: \"kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.757180 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.757163    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:04.757226 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.757205    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.757263 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.757232    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls podName:88cae88a-5ad3-4932-b27e-1a012e018d55 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:05.257213925 +0000 UTC m=+30.562975786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls") pod "dns-default-2rzs7" (UID: "88cae88a-5ad3-4932-b27e-1a012e018d55") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:04.757517 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.757494    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.759692 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.759672    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.760759 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.760740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.776301 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.776278    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 07:24:04.776584 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.776270    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 07:24:04.778437 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.776328    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 07:24:04.778437 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.776363    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 07:24:04.778437 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.776610    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\""
Nov 02 07:24:04.778437 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.777701    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc2t9"]
Nov 02 07:24:04.778948 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.778923    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh2r\" (UniqueName: \"kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.779071 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.779057    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndwz\" (UniqueName: \"kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:04.821525 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.821452    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:24:04.824203 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.824189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:04.847652 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.847627    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:24:04.851094 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.851067    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 07:24:04.858175 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.858149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.858294 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.858184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.858294 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.858277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jnp\" (UniqueName: \"kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.858403 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.858389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.858437 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.858415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.862711 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.862684    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 07:24:04.865752 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.865737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:04.876528 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.876509    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ztvhn\""
Nov 02 07:24:04.958694 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.958694 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jnp\" (UniqueName: \"kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.958819    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.958848    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:04.958905 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.958900    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:05.45887959 +0000 UTC m=+30.764641456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:04.959217 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.959217 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:04.958929    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:05.458914094 +0000 UTC m=+30.764675934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:04.959217 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.958923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:04.959217 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.959067    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:04.983208 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:04.983184    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jnp\" (UniqueName: \"kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:05.046640 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.046603    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"]
Nov 02 07:24:05.052925 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.052902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:05.059273 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.059250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.059411 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.059279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.059411 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.059370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.059528 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.059423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.059714 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.059698    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.060330 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.060296    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.060438 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.060419    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.071997 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.071944    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"]
Nov 02 07:24:05.076436 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.076417    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-48668\""
Nov 02 07:24:05.089360 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.089336    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.100237 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.100210    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 07:24:05.100237 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.100210    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:05.135057 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.135028    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:05.160536 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.160500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75sj\" (UniqueName: \"kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj\") pod \"migrator-66df5d496b-9jvf4\" (UID: \"522f311c-5aeb-455c-b170-d008487dabce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:05.170054 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.170026    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dfxs"]
Nov 02 07:24:05.174977 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:05.174952    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fa6161_e719_4d4b_8d83_5188300c9a4d.slice/crio-7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f WatchSource:0}: Error finding container 7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f: Status 404 returned error can't find the container with id 7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f
Nov 02 07:24:05.259591 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.259555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:05.259750 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.259729    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:05.261015 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.260987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:05.261134 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.261030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75sj\" (UniqueName: \"kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj\") pod \"migrator-66df5d496b-9jvf4\" (UID: \"522f311c-5aeb-455c-b170-d008487dabce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:05.261178 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.261139    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:05.261224 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.261205    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls podName:88cae88a-5ad3-4932-b27e-1a012e018d55 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.261186321 +0000 UTC m=+31.566948172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls") pod "dns-default-2rzs7" (UID: "88cae88a-5ad3-4932-b27e-1a012e018d55") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:05.272831 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.272814    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 07:24:05.272947 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.272924    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\""
Nov 02 07:24:05.272947 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.272939    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\""
Nov 02 07:24:05.273021 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.273010    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 07:24:05.281687 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.281665    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75sj\" (UniqueName: \"kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj\") pod \"migrator-66df5d496b-9jvf4\" (UID: \"522f311c-5aeb-455c-b170-d008487dabce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:05.365955 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.365873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:05.461936 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.461904    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:24:05.462497 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.462449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:05.462582 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.462492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:05.462582 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.462552    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.462693 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.462624    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.462693 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.462641    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.462619138 +0000 UTC m=+31.768380999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.462693 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:05.462674    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.462658665 +0000 UTC m=+31.768420529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.465336 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:05.465297    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53590114_7b6a_4223_8361_f82630bb6d5d.slice/crio-0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3 WatchSource:0}: Error finding container 0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3: Status 404 returned error can't find the container with id 0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3
Nov 02 07:24:05.502326 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.502280    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" event={"ID":"53590114-7b6a-4223-8361-f82630bb6d5d","Type":"ContainerStarted","Data":"0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3"}
Nov 02 07:24:05.503412 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.503389    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dfxs" event={"ID":"97fa6161-e719-4d4b-8d83-5188300c9a4d","Type":"ContainerStarted","Data":"7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f"}
Nov 02 07:24:05.688505 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.688432    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"]
Nov 02 07:24:05.692074 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:05.692047    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522f311c_5aeb_455c_b170_d008487dabce.slice/crio-72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb WatchSource:0}: Error finding container 72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb: Status 404 returned error can't find the container with id 72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb
Nov 02 07:24:05.975447 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:05.975375    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wwl6t"]
Nov 02 07:24:06.006195 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.006166    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wwl6t"]
Nov 02 07:24:06.006387 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.006286    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.019383 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.019239    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 07:24:06.019383 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.019328    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 07:24:06.019383 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.019327    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 07:24:06.019383 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.019378    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 07:24:06.019593 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.019398    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-977zc\""
Nov 02 07:24:06.068009 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.067967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.068182 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.068021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.068182 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.068116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk79w\" (UniqueName: \"kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.168640 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.168596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk79w\" (UniqueName: \"kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.168858 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.168767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.168858 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.168811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.169705 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.169674    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.171866 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.171842    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.192516 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.192489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk79w\" (UniqueName: \"kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.258992 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.258962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:06.269300 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.269271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:06.269433 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.269418    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:06.269496 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.269483    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls podName:88cae88a-5ad3-4932-b27e-1a012e018d55 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.269464426 +0000 UTC m=+33.575226267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls") pod "dns-default-2rzs7" (UID: "88cae88a-5ad3-4932-b27e-1a012e018d55") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:06.275590 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.275565    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 07:24:06.275590 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.275565    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 07:24:06.275801 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.275699    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\""
Nov 02 07:24:06.335023 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.334980    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:06.472195 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.472149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:06.472375 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.472209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:06.472456 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.472441    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:06.472517 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.472506    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.472488039 +0000 UTC m=+33.778249897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:06.472911 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.472895    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:06.472966 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:06.472947    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.47293146 +0000 UTC m=+33.778693304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:06.510070 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.509974    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4" event={"ID":"522f311c-5aeb-455c-b170-d008487dabce","Type":"ContainerStarted","Data":"72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb"}
Nov 02 07:24:06.654449 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:06.654407    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wwl6t"]
Nov 02 07:24:06.658180 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:06.658120    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78c693_2031_45f1_910a_c89b469c8b69.slice/crio-a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18 WatchSource:0}: Error finding container a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18: Status 404 returned error can't find the container with id a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18
Nov 02 07:24:07.514581 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:07.514554    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t" event={"ID":"bf78c693-2031-45f1-910a-c89b469c8b69","Type":"ContainerStarted","Data":"a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18"}
Nov 02 07:24:08.288138 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.288090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:08.288300 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.288231    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:08.288378 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.288300    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls podName:88cae88a-5ad3-4932-b27e-1a012e018d55 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.288276902 +0000 UTC m=+37.594038767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls") pod "dns-default-2rzs7" (UID: "88cae88a-5ad3-4932-b27e-1a012e018d55") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:08.490669 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.490587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:08.490669 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.490634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:08.490872 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.490717    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:08.490872 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.490753    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:08.490872 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.490797    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.490775875 +0000 UTC m=+37.796537716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:08.490872 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.490813    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.490806539 +0000 UTC m=+37.796568377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:08.519556 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.519510    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" event={"ID":"53590114-7b6a-4223-8361-f82630bb6d5d","Type":"ContainerStarted","Data":"5ffe369fa0fc40aceb601747c614102166f395af26e2b0c758c9012a732d4cc5"}
Nov 02 07:24:08.521407 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.521375    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4" event={"ID":"522f311c-5aeb-455c-b170-d008487dabce","Type":"ContainerStarted","Data":"70bca016df4daa10f25e3bfd4a0925895f569027cc5e64422dae99dd9a41756b"}
Nov 02 07:24:08.556128 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.556080    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" podStartSLOduration=1.8427983860000001 podStartE2EDuration="4.556067194s" totalImagesPullingTime="2.713268808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:08.555680594 +0000 UTC m=+33.861442456" watchObservedRunningTime="2026-11-02 07:24:08.556067194 +0000 UTC m=+33.861829054"
Nov 02 07:24:08.590368 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.590297    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4" podStartSLOduration=1.7933857149999999 podStartE2EDuration="3.590280383s" totalImagesPullingTime="1.796894668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:08.59020092 +0000 UTC m=+33.895962785" watchObservedRunningTime="2026-11-02 07:24:08.590280383 +0000 UTC m=+33.896042242"
Nov 02 07:24:08.995569 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.995531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:08.995796 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.995585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:08.995796 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.995689    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:08.995796 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:08.995751    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs podName:d7a903fa-2c1d-47ed-8701-1c220c920b57 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:40.995736874 +0000 UTC m=+66.301498732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs") pod "network-metrics-daemon-qnkkt" (UID: "d7a903fa-2c1d-47ed-8701-1c220c920b57") : secret "metrics-daemon-secret" not found
Nov 02 07:24:08.999184 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:08.999132    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:09.096335 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.096247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:09.099004 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.098977    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:09.190468 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.190423    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:09.287822 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.287488    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:09.525711 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.525677    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dfxs" event={"ID":"97fa6161-e719-4d4b-8d83-5188300c9a4d","Type":"ContainerStarted","Data":"f0ed746f0953014877e00cd69367aa7831936340bc10741f5c59619173eaa614"}
Nov 02 07:24:09.532230 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.532136    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brmn4"]
Nov 02 07:24:09.534685 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:09.534657    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3daa4be_adf6_4e9f_8307_98e3c0d180ec.slice/crio-6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82 WatchSource:0}: Error finding container 6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82: Status 404 returned error can't find the container with id 6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82
Nov 02 07:24:09.564650 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.564602    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9dfxs" podStartSLOduration=1.9616458140000002 podStartE2EDuration="5.564589587s" totalImagesPullingTime="3.602943773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:09.563955605 +0000 UTC m=+34.869717466" watchObservedRunningTime="2026-11-02 07:24:09.564589587 +0000 UTC m=+34.870351446"
Nov 02 07:24:09.602533 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:09.602502    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq8m6"]
Nov 02 07:24:09.744402 ip-10-0-137-156 kubenswrapper[2573]: W1102 07:24:09.744368    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc523525_bb72_4547_a2dc_c77957b278b0.slice/crio-999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a WatchSource:0}: Error finding container 999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a: Status 404 returned error can't find the container with id 999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a
Nov 02 07:24:10.531040 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:10.531004    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq8m6" event={"ID":"dc523525-bb72-4547-a2dc-c77957b278b0","Type":"ContainerStarted","Data":"999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a"}
Nov 02 07:24:10.533405 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:10.533371    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t" event={"ID":"bf78c693-2031-45f1-910a-c89b469c8b69","Type":"ContainerStarted","Data":"0fe2579a846d58cf46ed82398fb36e719fa4379230938cbb355f67b52a00269b"}
Nov 02 07:24:10.534715 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:10.534690    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-brmn4" event={"ID":"a3daa4be-adf6-4e9f-8307-98e3c0d180ec","Type":"ContainerStarted","Data":"6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82"}
Nov 02 07:24:10.569268 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:10.569202    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t" podStartSLOduration=2.448699867 podStartE2EDuration="5.569182592s" totalImagesPullingTime="3.120482725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:10.568819997 +0000 UTC m=+35.874581858" watchObservedRunningTime="2026-11-02 07:24:10.569182592 +0000 UTC m=+35.874944455"
Nov 02 07:24:10.863935 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:10.863785    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97v78_71b02e54-cab8-41d7-909c-7f0cbe5eab71/dns-node-resolver/0.log"
Nov 02 07:24:12.261118 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.261082    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mldbl_903d010c-458d-4c3a-8c94-8bf685cb6ef3/node-ca/0.log"
Nov 02 07:24:12.328407 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.328367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:12.328585 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:12.328537    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:12.328639 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:12.328602    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls podName:88cae88a-5ad3-4932-b27e-1a012e018d55 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:20.328589376 +0000 UTC m=+45.634351215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls") pod "dns-default-2rzs7" (UID: "88cae88a-5ad3-4932-b27e-1a012e018d55") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:12.530735 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.530097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:12.530735 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.530156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:12.530735 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:12.530259    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:12.530735 ip-10-0-137-156 kubenswrapper[2573]: E1102 07:24:12.530337    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:20.530319971 +0000 UTC m=+45.836081823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:12.530982 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.530761    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:12.661377 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:12.661341    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9dfxs_97fa6161-e719-4d4b-8d83-5188300c9a4d/serve-healthcheck-canary/0.log"
Nov 02 07:24:13.061975 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:13.061942    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-9jvf4_522f311c-5aeb-455c-b170-d008487dabce/migrator/0.log"
Nov 02 07:24:14.546556 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:14.546525    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq8m6" event={"ID":"dc523525-bb72-4547-a2dc-c77957b278b0","Type":"ContainerStarted","Data":"7bd5d50c1cae79bfd3eb9d575888fe22d37e0072397b838ce2e5e3116e66ae06"}
Nov 02 07:24:14.547041 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:14.546650    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:14.585542 ip-10-0-137-156 kubenswrapper[2573]: I1102 07:24:14.585496    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xq8m6" podStartSLOduration=35.772485459 podStartE2EDuration="39.585483922s" totalImagesPullingTime="3.812998463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:14.584566606 +0000 UTC m=+39.890328467" watchObservedRunningTime="2026-11-02 07:24:14.585483922 +0000 UTC m=+39.891245782"
Nov 02 07:24:15.182994 ip-10-0-137-156 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 07:24:15.195676 ip-10-0-137-156 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 07:24:15.195848 ip-10-0-137-156 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 07:24:15.195897 ip-10-0-137-156 systemd[1]: kubelet.service: Consumed 2.442s CPU time.
Nov 02 07:24:15.207854 ip-10-0-137-156 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 07:24:15.293092 ip-10-0-137-156 kubenswrapper[4515]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:15.293092 ip-10-0-137-156 kubenswrapper[4515]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 07:24:15.293092 ip-10-0-137-156 kubenswrapper[4515]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:15.293751 ip-10-0-137-156 kubenswrapper[4515]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:15.295987 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.295894    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.295987 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.295986    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.295987 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.295990    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.295994    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.295997    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296000    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296003    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296005    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296008    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296011    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296014    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296017    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296020    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296023    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296025    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296028    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296031    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296034    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296037    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296040    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296046    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296049    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.296131 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296052    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296055    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296058    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296060    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296064    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296066    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296069    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296072    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296075    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296078    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296080    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296085    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296089    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296093    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296096    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296099    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296102    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296106    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296108    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.296688 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296111    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296114    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296117    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296120    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296122    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296125    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296128    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296131    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296134    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296137    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296141    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296143    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296146    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296149    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296152    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296154    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296157    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296174    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296178    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296181    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.297169 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296184    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296187    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296190    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296193    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296196    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296199    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296202    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296205    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296207    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296210    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296213    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296216    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296219    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296221    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296224    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296227    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296230    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296232    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.297687 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296235    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296238    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296241    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296244    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296247    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296249    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296252    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296255    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296262    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296265    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296268    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296271    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296274    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296276    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296279    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296282    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296285    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296287    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296291    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296293    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.298128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296296    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296301    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296304    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296333    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296338    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296341    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.296343    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297182    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297189    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297193    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297196    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297199    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297202    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297205    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297208    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297216    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297219    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297222    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297225    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297227    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.298647 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297232    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297235    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297238    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297241    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297244    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297247    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297249    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297252    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297255    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297257    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297260    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297263    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297265    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297269    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297272    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297275    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297277    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297280    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297283    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297286    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.299139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297288    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297291    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297294    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297296    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297299    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297301    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297304    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297324    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297337    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297341    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297344    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297348    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297352    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297355    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297357    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297360    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297363    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297365    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297368    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.299651 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297371    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297374    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297376    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297379    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297382    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297385    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297388    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297390    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297393    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297396    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297399    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297402    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297404    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297407    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297410    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297413    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297417    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297421    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297424    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.300121 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297427    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297430    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297432    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297441    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297445    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297448    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297452    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297455    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297458    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297460    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297463    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297466    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297469    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297471    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297474    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297477    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297480    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297482    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297485    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297488    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.300609 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297490    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297493    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297496    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297498    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297501    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297503    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297506    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297509    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297511    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297514    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297517    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297520    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297522    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297525    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.297527    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298069    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298075    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298078    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298083    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.301108 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298086    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298089    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298091    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298094    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298097    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298100    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298102    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298105    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298108    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298111    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298114    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298116    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298119    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298121    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298124    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298127    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298130    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298132    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298137    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.301580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298141    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298144    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298147    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298152    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298155    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298158    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298160    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298163    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298166    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298169    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298172    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298175    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298178    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298181    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298184    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298187    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298190    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298193    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298196    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.302098 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298198    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298201    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298204    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298206    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298209    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298212    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298214    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298217    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298220    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298222    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298225    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298227    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298230    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298233    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298236    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298238    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298242    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298245    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298248    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298250    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.302611 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298253    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298257    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298262    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298265    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298270    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298273    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298277    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298279    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298283    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298285    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298288    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298291    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298294    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298296    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298299    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298302    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298319    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298323    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298328    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298332    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.303118 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298335    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298338    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298341    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298344    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298347    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298349    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298352    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298355    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298359    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298362    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298365    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298379    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298382    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298385    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298388    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298390    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298393    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298396    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298399    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298402    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.303636 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298406    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298409    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298411    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.298414    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298513    4515 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298525    4515 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298534    4515 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298540    4515 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298545    4515 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298549    4515 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298554    4515 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298558    4515 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298562    4515 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298565    4515 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298568    4515 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298572    4515 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298575    4515 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298578    4515 flags.go:64] FLAG: --cgroup-root=""
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298581    4515 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298584    4515 flags.go:64] FLAG: --client-ca-file=""
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298591    4515 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298594    4515 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298599    4515 flags.go:64] FLAG: --cluster-domain=""
Nov 02 07:24:15.304223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298602    4515 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298605    4515 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298608    4515 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298612    4515 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298616    4515 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298619    4515 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298622    4515 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298626    4515 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298629    4515 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298632    4515 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298635    4515 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298639    4515 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298641    4515 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298646    4515 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298649    4515 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298652    4515 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298655    4515 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298659    4515 flags.go:64] FLAG: --enable-server="true"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298662    4515 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298667    4515 flags.go:64] FLAG: --event-burst="100"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298670    4515 flags.go:64] FLAG: --event-qps="50"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298674    4515 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298678    4515 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298681    4515 flags.go:64] FLAG: --eviction-hard=""
Nov 02 07:24:15.304794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298686    4515 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298689    4515 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298692    4515 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298695    4515 flags.go:64] FLAG: --eviction-soft=""
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298699    4515 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298703    4515 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298706    4515 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298709    4515 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298712    4515 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298715    4515 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298718    4515 flags.go:64] FLAG: --feature-gates=""
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298725    4515 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298729    4515 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298732    4515 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298735    4515 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298738    4515 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298742    4515 flags.go:64] FLAG: --help="false"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298745    4515 flags.go:64] FLAG: --hostname-override="ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298749    4515 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298752    4515 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298755    4515 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298759    4515 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298762    4515 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298765    4515 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 07:24:15.305434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298768    4515 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298771    4515 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298774    4515 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298777    4515 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298780    4515 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298785    4515 flags.go:64] FLAG: --kube-reserved=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298788    4515 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298791    4515 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298794    4515 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298797    4515 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298800    4515 flags.go:64] FLAG: --lock-file=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298803    4515 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298806    4515 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298811    4515 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298816    4515 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298819    4515 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298823    4515 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298826    4515 flags.go:64] FLAG: --logging-format="text"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298829    4515 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298833    4515 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298835    4515 flags.go:64] FLAG: --manifest-url=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298838    4515 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298843    4515 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298846    4515 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298850    4515 flags.go:64] FLAG: --max-pods="110"
Nov 02 07:24:15.306330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298855    4515 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298858    4515 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298862    4515 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298865    4515 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298868    4515 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298871    4515 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298874    4515 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298881    4515 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298884    4515 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298888    4515 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298891    4515 flags.go:64] FLAG: --pod-cidr=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298894    4515 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298898    4515 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298901    4515 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298905    4515 flags.go:64] FLAG: --port="10250"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298908    4515 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298911    4515 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01514a35ac383ddcf"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298915    4515 flags.go:64] FLAG: --qos-reserved=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298918    4515 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298921    4515 flags.go:64] FLAG: --register-node="true"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298925    4515 flags.go:64] FLAG: --register-with-taints=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298929    4515 flags.go:64] FLAG: --registry-burst="10"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298932    4515 flags.go:64] FLAG: --registry-qps="5"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298935    4515 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298938    4515 flags.go:64] FLAG: --reserved-memory=""
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298943    4515 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 07:24:15.307036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298946    4515 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298949    4515 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298952    4515 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298955    4515 flags.go:64] FLAG: --runonce="false"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298958    4515 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298962    4515 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298965    4515 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298968    4515 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298972    4515 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298975    4515 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298978    4515 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298982    4515 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298985    4515 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298988    4515 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298991    4515 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298994    4515 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.298997    4515 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299000    4515 flags.go:64] FLAG: --system-cgroups=""
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299004    4515 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299010    4515 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299013    4515 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299016    4515 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299021    4515 flags.go:64] FLAG: --tls-min-version=""
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299024    4515 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299027    4515 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 07:24:15.307709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299030    4515 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299035    4515 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299038    4515 flags.go:64] FLAG: --v="2"
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299043    4515 flags.go:64] FLAG: --version="false"
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299048    4515 flags.go:64] FLAG: --vmodule=""
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299052    4515 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299055    4515 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299193    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299196    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299199    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299203    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299206    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299208    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299211    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299214    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299217    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299220    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299223    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299225    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299228    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299231    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.308514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299234    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299237    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299239    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299243    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299249    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299258    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299261    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299264    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299267    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299270    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299272    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299275    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299278    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299281    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299284    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299287    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299289    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299292    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299295    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.309258 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299298    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299301    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299304    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299319    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299322    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299325    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299328    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299331    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299340    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299344    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299348    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299351    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299354    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299357    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299359    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299362    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299365    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299369    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299372    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299375    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.309928 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299377    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299380    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299383    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299386    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299389    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299392    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299394    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299397    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299400    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299403    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299405    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299408    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299411    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299428    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299432    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299435    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299438    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299441    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299444    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.310752 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299447    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299450    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299458    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299461    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299464    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299467    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299469    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299472    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299475    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299478    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299482    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299484    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299487    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299490    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299493    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299496    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299498    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299502    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299504    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.311511 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299507    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299510    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299513    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299515    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299518    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299521    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299524    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299526    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299529    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299532    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299534    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299537    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299540    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299542    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299545    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.299571    4515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:15.312146 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299613    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299618    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299621    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299625    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299630    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299633    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299637    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299641    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299645    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299648    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299651    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299654    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299657    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299659    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299662    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299665    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299668    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299670    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299673    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.312890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299676    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299679    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299681    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299684    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299687    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299690    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299693    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299695    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299698    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299700    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299703    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299706    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299710    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299713    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299724    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299728    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299731    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299735    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299737    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299740    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.313650 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299743    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299747    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299749    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299752    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299755    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299758    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299761    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299764    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299768    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299770    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299773    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299776    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299779    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299781    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299784    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299787    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299789    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299792    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299795    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.314363 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299797    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299800    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299803    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299805    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299808    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299811    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299814    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299816    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299825    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299828    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299832    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299834    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299837    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299840    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299844    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299847    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299849    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299852    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299855    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.314911 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299858    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299860    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299863    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299865    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299868    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299871    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299873    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299876    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299879    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299881    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299884    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299887    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299889    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299892    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299895    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299897    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299900    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299903    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299905    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299908    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.315478 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299911    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299914    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299918    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299922    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299925    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299927    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299930    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299936    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.299939    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.303967    4515 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.303986    4515 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.304006    4515 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.304010    4515 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304067    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304073    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304077    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304080    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304082    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304087    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.315991 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304091    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304094    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304097    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304100    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304103    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304105    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304108    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304111    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304114    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304117    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304119    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304122    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304125    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304128    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304130    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304133    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304136    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304138    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304141    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.316477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304144    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304146    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304149    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304154    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304159    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304163    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304167    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304170    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304173    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304176    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304178    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304181    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304183    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304186    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304189    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304191    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304194    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304197    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304199    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304202    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.316948 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304205    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304207    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304210    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304213    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304215    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304218    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304221    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304223    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304226    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304229    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304232    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304235    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304238    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304240    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304243    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304246    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304249    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304253    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304255    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.317473 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304258    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304261    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304263    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304266    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304269    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304271    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304274    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304276    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304279    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304282    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304285    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304287    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304290    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304292    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304295    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304298    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304300    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304303    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304327    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.317979 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304331    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304334    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304337    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304340    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304343    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304345    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304348    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304351    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304354    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304357    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304360    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304363    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304366    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304369    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304372    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304374    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304377    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304380    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304382    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304385    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.318519 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304388    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304390    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304393    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.304421    4515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304557    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304562    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304565    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304568    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304570    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304573    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304575    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304578    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304581    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304583    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304586    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304589    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.319016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304594    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304598    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304601    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304604    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304607    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304610    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304612    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304615    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304617    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304621    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304623    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304626    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304629    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304632    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304634    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304637    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304640    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304643    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304645    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304648    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.319526 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304650    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304653    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304655    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304658    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304661    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304663    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304666    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304668    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304671    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304674    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304676    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304679    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304682    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304685    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304687    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304690    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304693    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304695    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304698    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.320008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304700    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304703    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304706    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304709    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304712    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304714    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304717    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304720    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304722    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304725    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304728    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304730    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304733    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304736    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304738    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304741    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304744    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304746    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304749    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304751    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.320485 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304754    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304757    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304759    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304762    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304765    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304774    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304777    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304779    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304782    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304785    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304787    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304790    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304793    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304796    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304799    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304801    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304805    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304807    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304810    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304813    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.321002 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304818    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304821    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304824    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304826    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304829    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304831    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304834    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304837    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304839    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304843    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304846    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304849    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304852    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304854    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304857    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.304880    4515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:15.321506 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304910    4515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304914    4515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304917    4515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304919    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304922    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304925    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304927    4515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304930    4515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304932    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304935    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304938    4515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304940    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304943    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304946    4515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304948    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304951    4515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304955    4515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304958    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304961    4515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:15.321900 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304963    4515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304966    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304969    4515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304971    4515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304974    4515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304977    4515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304979    4515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304983    4515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304985    4515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304988    4515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304991    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304993    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304996    4515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.304998    4515 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305001    4515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305004    4515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305006    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305009    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305012    4515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305014    4515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:15.322407 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305017    4515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305019    4515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305022    4515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305025    4515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305027    4515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305030    4515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305032    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305035    4515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305038    4515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305050    4515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305053    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305056    4515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305059    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305062    4515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305064    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305067    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305070    4515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305073    4515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305075    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305078    4515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:15.322881 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305080    4515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305083    4515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305086    4515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305088    4515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305091    4515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305094    4515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305096    4515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305099    4515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305101    4515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305104    4515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305107    4515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305109    4515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305112    4515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305116    4515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305119    4515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305122    4515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305125    4515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305127    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305130    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:15.323438 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305133    4515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305136    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305141    4515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305144    4515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305147    4515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305150    4515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305152    4515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305155    4515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305157    4515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305160    4515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305163    4515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305165    4515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305170    4515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305173    4515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305176    4515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305179    4515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305183    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305186    4515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305189    4515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:15.323893 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305192    4515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305195    4515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305198    4515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305201    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305203    4515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305206    4515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305209    4515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305212    4515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:15.305215    4515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.305272    4515 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306246    4515 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306303    4515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306499    4515 server.go:1014] "Starting client certificate rotation"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306641    4515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306689    4515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:35 +0000 UTC" deadline="2028-05-27 08:32:20.123545115 +0000 UTC"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306723    4515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13729h8m4.816823875s"
Nov 02 07:24:15.324570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.306918    4515 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:24:15.325246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.308124    4515 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:24:15.325246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.311456    4515 log.go:25] "Validated CRI v1 runtime API"
Nov 02 07:24:15.325246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.313593    4515 log.go:25] "Validated CRI v1 image API"
Nov 02 07:24:15.325246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.314436    4515 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 07:24:15.325246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.316086    4515 fs.go:135] Filesystem UUIDs: map[0021f34d-aa3b-4fca-8f1d-ed3c2cc5f08f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c70c9398-4a38-48ad-9a50-3cb2f9a7fe2a:/dev/nvme0n1p4]
Nov 02 07:24:15.325519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.316113    4515 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/763de3ed6b90822725f6ce67d1fd4b84ae7caae15dfaef9c67f6d0716423c94e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/763de3ed6b90822725f6ce67d1fd4b84ae7caae15dfaef9c67f6d0716423c94e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4468fdc21b1dacf7baf50dcc9c5f6e811fc39441acada88892cfb7703dc006f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4468fdc21b1dacf7baf50dcc9c5f6e811fc39441acada88892cfb7703dc006f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f836f8b-5b38-4697-b577-46cc16c02f12/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0f836f8b-5b38-4697-b577-46cc16c02f12/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2780dc0b-f641-45c0-86f8-769a72bf589c/volumes/kubernetes.io~projected/kube-api-access-hsnh5:{mountpoint:/var/lib/kubelet/pods/2780dc0b-f641-45c0-86f8-769a72bf589c/volumes/kubernetes.io~projected/kube-api-access-hsnh5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27a575b2-ee3c-4647-b1a6-8b29faed0a41/volumes/kubernetes.io~projected/kube-api-access-nvgsr:{mountpoint:/var/lib/kubelet/pods/27a575b2-ee3c-4647-b1a6-8b29faed0a41/volumes/kubernetes.io~projected/kube-api-access-nvgsr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1abc9-0c41-4337-b497-b4ddff25663d/volumes/kubernetes.io~projected/kube-api-access-qd7zj:{mountpoint:/var/lib/kubelet/pods/3ec1abc9-0c41-4337-b497-b4ddff25663d/volumes/kubernetes.io~projected/kube-api-access-qd7zj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43a44cf5-6890-4936-861f-7f26de059789/volumes/kubernetes.io~projected/kube-api-access-5ghdw:{mountpoint:/var/lib/kubelet/pods/43a44cf5-6890-4936-861f-7f26de059789/volumes/kubernetes.io~projected/kube-api-access-5ghdw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~projected/kube-api-access-cl6km:{mountpoint:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~projected/kube-api-access-cl6km major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/522f311c-5aeb-455c-b170-d008487dabce/volumes/kubernetes.io~projected/kube-api-access-p75sj:{mountpoint:/var/lib/kubelet/pods/522f311c-5aeb-455c-b170-d008487dabce/volumes/kubernetes.io~projected/kube-api-access-p75sj major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53590114-7b6a-4223-8361-f82630bb6d5d/volumes/kubernetes.io~projected/kube-api-access-vmqzf:{mountpoint:/var/lib/kubelet/pods/53590114-7b6a-4223-8361-f82630bb6d5d/volumes/kubernetes.io~projected/kube-api-access-vmqzf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71b02e54-cab8-41d7-909c-7f0cbe5eab71/volumes/kubernetes.io~projected/kube-api-access-2lbm9:{mountpoint:/var/lib/kubelet/pods/71b02e54-cab8-41d7-909c-7f0cbe5eab71/volumes/kubernetes.io~projected/kube-api-access-2lbm9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~projected/kube-api-access-dz6js:{mountpoint:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~projected/kube-api-access-dz6js major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88cae88a-5ad3-4932-b27e-1a012e018d55/volumes/kubernetes.io~projected/kube-api-access-7ndwz:{mountpoint:/var/lib/kubelet/pods/88cae88a-5ad3-4932-b27e-1a012e018d55/volumes/kubernetes.io~projected/kube-api-access-7ndwz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/903d010c-458d-4c3a-8c94-8bf685cb6ef3/volumes/kubernetes.io~projected/kube-api-access-85d7r:{mountpoint:/var/lib/kubelet/pods/903d010c-458d-4c3a-8c94-8bf685cb6ef3/volumes/kubernetes.io~projected/kube-api-access-85d7r major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~projected/kube-api-access-xnh2r:{mountpoint:/var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~projected/kube-api-access-xnh2r major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3daa4be-adf6-4e9f-8307-98e3c0d180ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a3daa4be-adf6-4e9f-8307-98e3c0d180ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~projected/kube-api-access-lk79w:{mountpoint:/var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~projected/kube-api-access-lk79w major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a903fa-2c1d-47ed-8701-1c220c920b57/volumes/kubernetes.io~projected/kube-api-access-hs9cw:{mountpoint:/var/lib/kubelet/pods/d7a903fa-2c1d-47ed-8701-1c220c920b57/volumes/kubernetes.io~projected/kube-api-access-hs9cw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc523525-bb72-4547-a2dc-c77957b278b0/volumes/kubernetes.io~projected/kube-api-access-4dlg9:{mountpoint:/var/lib/kubelet/pods/dc523525-bb72-4547-a2dc-c77957b278b0/volumes/kubernetes.io~projected/kube-api-access-4dlg9 major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e73491b7-617f-4eb1-b094-69a5db32b449/volumes/kubernetes.io~projected/kube-api-access-g2jnp:{mountpoint:/var/lib/kubelet/pods/e73491b7-617f-4eb1-b094-69a5db32b449/volumes/kubernetes.io~projected/kube-api-access-g2jnp major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/24cdc4a60d95dbab11e1bfd33799a571c09ec684c51fe4ea8aba84309bc08e74/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/350b124269368f873d1b1da6f36561210dbb493b0256954e98cb702434c8bdd7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b4256772394221879c0c9016565000d3695dda55787ec523cd9d59c09421c158/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f89c3892871472878c3131fc24c4aadc5f1260c66dd6bfe0341cf53b4790c0a0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6362713bb3ed7825e6074ec0e5a18edc3ad7fe78b7f08d64f990a0ec97843f8b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5d6e00e9aad85ee9329a9bd538db84517a55fde72512e56b88dbdd87261fa244/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5645e679aa9594c105c65326b0a93c90d7b9c8b1c3b6ea174cbb2b2b3b3449ad/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/931e765dda3bf40cca05ecd8554fdf4cd12b9d8b1c5dd9254bf3e1096e938fd3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1046a09b5f27cce94593c8ab3984d5bc97f3dbfedfd40e43a8710e7b77f05029/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/25fc4f8a0d810ea36d58c85eaa72062c884970e5af4f7d916191e885ac18794b/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6f4f0d8282b0287b9934e65e3a61cb64c19923447826af00be2d143da4c6ea05/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/6f0257c0efb89227ddb3939cf6fe8104f2a7247c1cde77925946222ec0c752c5/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a25965fb00e1c3828637b82d9561ed0f59bf9fab60deb8f5f0cbc4a2326513b6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/10894f225143d5393c7c868ae9b7dc8ccb68946cb4226c77412389bcf20dc53b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e4eb369b616aa3cf8f4fcb374dabde4bedaf194aee493cc6928fff7ae717d4f3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4d083e8f4e12b91f50b0603507f8ce17686c93f62eddde220e77a4eb44ed2227/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a431db1b9c667292a9c398ebe3c7652312722f2c4ea15b9f816f038048579900/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f77123722981600edd899df0945fbd473be1a3ebe31e839a22a33656d2297160/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6d5cbb15949d9dd3d24cc1ca60542e2742e7aa1dc30f4b0e4f9c46f6ec074a16/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/596aa4a76c755df4fafbc33e052e7deff7a771c3b3f74812912ccc3e3005861c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/895b59f1a3a11010c5aa52372e241e3ec20a0ca06df69c53673f2c7deb3345c3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/c999a4bb2b2f4da33d398ae45f20a32444e85028be7b9fb925e6a416a0f31c5f/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a47a854b7e2314a683d6ed85f2eca310d01d2462cf6b68320a894db3625d2938/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/39f7915155b89a9c3294dc0a99b882c933527f7e0c5b6866649a4821fd1dbfb0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/4ded637872c45f045b67d19d26fcad9777c0735d5b058da0b9af459633299db0/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/ecdc26ba4421f866e1cad71737e6e44a10f08fb51db2597fa1eca6e5b88337d8/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/84ca9f922367571f46158c2edd883a8a6b9f71e4f27f8eb549efd3f3050ab223/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/9f55f726de25cd86232ae7b60786be4a0a17ed7659e68504f49cd24858db45d5/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/679727bcfc0c0c773eea373c07ad684005868ed5162dc377422444b918944a14/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/e924764c0a3accab2b33dc2884d7c46004c30112ca00acfd52527208319faf6f/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/3215c79719e98dffa0bdbcac0a4df3834cc08be956b511264800ec8bc27975b9/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/509bd242fecffd1aed29ee480345468f9888b16eec7a66acad4f9ea08daf9e64/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/2ee3c8224e7e5f5b517a9e919005bfdb13d85bc554fcb58faa04f2f46db82ad4/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/0a45befa5b909648556a713376bf0f81cba83a54ff8ba4f57e607f63efab55e5/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/118d77bbba8d708d9ff8bd88f3f817787f029b2bf974d70fe57de685af4fc809/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f26be4d3f11ab0973d8681e904e937099a8235e01cda7cf425eaacd576e1b8f2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d74c573850d0840c0fe4206aa05caf08b6e59b79c9d89287b1ab9dd3bfd26789/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/3c73642332509796f2d94ae7898d2ee8625171405f80177c2e55a3b78dd287d8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fc539f1489aad6d28380b3d51d0d7ba0a37b10e15ad98a4acd381732e43adda8/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e67ba914eacc728b3fc4317db7d6f8055cb23dedaacefdae5c1f820244d5d644/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/17a70376d6a57dbfd19e47abeb92de90ccb3bd19c6bbad5184783e3eb04f7051/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7c172cc2404a0fb7d21c52d2754f32b87e2b0569eb77fdd94e809853558d07d4/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6421c0e1cd35de4072873bed3975414717667e1817519a00a5d402a757aaa7c9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.322758    4515 manager.go:217] Machine: {Timestamp:2026-11-02 07:24:15.322266251 +0000 UTC m=+0.084703148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099415 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21422a3dfcebed0dea53ef19b98ba7 SystemUUID:ec21422a-3dfc-ebed-0dea-53ef19b98ba7 BootID:024b276f-4117-4fb0-b70f-0d0c3f42df7a Filesystems:[{Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/522f311c-5aeb-455c-b170-d008487dabce/volumes/kubernetes.io~projected/kube-api-access-p75sj DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6acccf5f2c914e420d4b0da0c15443c356637ed8955c50ccafce5193f7aa9d82/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~projected/kube-api-access-cl6km DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72a70dd7a79b7a3e56d0be289c564d76e999643d19c7fac8e9a789f5cadf4ebb/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1abc9-0c41-4337-b497-b4ddff25663d/volumes/kubernetes.io~projected/kube-api-access-qd7zj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c92d35ec0f1e3eec477a2f4315971c953d314cabf209e3c3b1e130dace1a35dc/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53590114-7b6a-4223-8361-f82630bb6d5d/volumes/kubernetes.io~projected/kube-api-access-vmqzf DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~projected/kube-api-access-xnh2r DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88cae88a-5ad3-4932-b27e-1a012e018d55/volumes/kubernetes.io~projected/kube-api-access-7ndwz DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/903d010c-458d-4c3a-8c94-8bf685cb6ef3/volumes/kubernetes.io~projected/kube-api-access-85d7r DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~projected/kube-api-access-dz6js DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~projected/kube-api-access-lk79w DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc523525-bb72-4547-a2dc-c77957b278b0/volumes/kubernetes.io~projected/kube-api-access-4dlg9 DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2780dc0b-f641-45c0-86f8-769a72bf589c/volumes/kubernetes.io~projected/kube-api-access-hsnh5 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27a575b2-ee3c-4647-b1a6-8b29faed0a41/volumes/kubernetes.io~projected/kube-api-access-nvgsr DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d47aed7b58c6d2ea6c278ec251fb25e641383317fdfe8a5d83e3a5782fceb07/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d4119a7ee989d11ec07040cd099972d96d95e69a28735d12b86441c0471cc7f/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43a44cf5-6890-4936-861f-7f26de059789/volumes/kubernetes.io~projected/kube-api-access-5ghdw DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f08491cf3ca3bc4119731acd9e2b20625dd863221a8b62221f8c437a981505f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d98a9be-246b-45a0-b64c-71119a497445/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cde4dc7b752caf678a9207865567f1011f21e3ad5cab0a1d1ac87b6b2659041/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d12867c37f301f7032989ed5f26c7203e58771a2395e853dac7d62581e519830/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/763de3ed6b90822725f6ce67d1fd4b84ae7caae15dfaef9c67f6d0716423c94e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97fa6161-e719-4d4b-8d83-5188300c9a4d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a903fa-2c1d-47ed-8701-1c220c920b57/volumes/kubernetes.io~projected/kube-api-access-hs9cw DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4468fdc21b1dacf7baf50dcc9c5f6e811fc39441acada88892cfb7703dc006f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8614740298c031547051c45be7926ced2bb6b272f9d37cc53e56abb2243bf18/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f836f8b-5b38-4697-b577-46cc16c02f12/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71b02e54-cab8-41d7-909c-7f0cbe5eab71/volumes/kubernetes.io~projected/kube-api-access-2lbm9 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9db7977965135ce0f9c4d6ec6395ec199b04d994a1ad06a7ea9e46b26f40b566/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e73491b7-617f-4eb1-b094-69a5db32b449/volumes/kubernetes.io~projected/kube-api-access-g2jnp DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/999640291d46d35b21a76aaeafaacc2790933f81a5835db9105293001692163a/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb95b14-6d7c-4241-b22a-dd67fa387dd3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3daa4be-adf6-4e9f-8307-98e3c0d180ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf78c693-2031-45f1-910a-c89b469c8b69/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0410ad40cfac516 MacAddress:ea:49:f8:e6:37:e9 Speed:10000 Mtu:8901} {Name:6acccf5f2c914e4 MacAddress:6e:78:23:59:64:29 Speed:10000 Mtu:8901} {Name:72a70dd7a79b7a3 MacAddress:de:4c:87:18:5d:33 Speed:10000 Mtu:8901} {Name:7d4119a7ee989d1 MacAddress:a6:09:40:96:d5:84 Speed:10000 Mtu:8901} {Name:999640291d46d35 MacAddress:4a:f4:1e:a6:a3:0a Speed:10000 Mtu:8901} {Name:a8614740298c031 MacAddress:2e:c2:d6:96:51:4f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:d3:be:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:52:e1:7f:9c:3a:fe Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:d3:be:3f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:02:25:f3:39:0b:c8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:af:33:48:57:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323068    4515 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323118    4515 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323446    4515 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323473    4515 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323609    4515 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323617    4515 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323628    4515 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323642    4515 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323728    4515 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.323900    4515 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324026    4515 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324060    4515 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324074    4515 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324084    4515 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324094    4515 kubelet.go:405] "Adding apiserver pod source"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324103    4515 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324775    4515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.324788    4515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:24:15.325751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.325354    4515 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 07:24:15.327541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327523    4515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 07:24:15.327541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327535    4515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:24:15.327679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327637    4515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 07:24:15.327855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327842    4515 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 07:24:15.327909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327871    4515 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 07:24:15.327979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327969    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.327993    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328000    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328006    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328014    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328022    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 07:24:15.328030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328029    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328035    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328042    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328047    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328063    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328072    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328098    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 07:24:15.328334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.328106    4515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 07:24:15.331563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.331547    4515 server.go:1263] "Started kubelet"
Nov 02 07:24:15.332770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.332098    4515 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 07:24:15.332770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.332203    4515 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 07:24:15.332770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.332709    4515 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 07:24:15.332402 ip-10-0-137-156 systemd[1]: Started Kubernetes Kubelet.
Nov 02 07:24:15.333912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.331627    4515 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 07:24:15.335633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.335361    4515 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 07:24:15.336485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.335885    4515 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 07:24:15.338765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.338110    4515 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 07:24:15.338765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.338125    4515 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 07:24:15.338765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.335922    4515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 07:24:15.338765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.338399    4515 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 07:24:15.338765 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.338439    4515 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-156.ec2.internal\" not found"
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340202    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb95b14-6d7c-4241-b22a-dd67fa387dd3" volumeName="kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340236    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cae88a-5ad3-4932-b27e-1a012e018d55" volumeName="kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340250    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2780dc0b-f641-45c0-86f8-769a72bf589c" volumeName="kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340262    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a575b2-ee3c-4647-b1a6-8b29faed0a41" volumeName="kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340274    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53590114-7b6a-4223-8361-f82630bb6d5d" volumeName="kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340284    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53590114-7b6a-4223-8361-f82630bb6d5d" volumeName="kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340295    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97fa6161-e719-4d4b-8d83-5188300c9a4d" volumeName="kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340436    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a903fa-2c1d-47ed-8701-1c220c920b57" volumeName="kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340452    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cae88a-5ad3-4932-b27e-1a012e018d55" volumeName="kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340465    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="903d010c-458d-4c3a-8c94-8bf685cb6ef3" volumeName="kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340477    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97fa6161-e719-4d4b-8d83-5188300c9a4d" volumeName="kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340488    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d98a9be-246b-45a0-b64c-71119a497445" volumeName="kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340500    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d98a9be-246b-45a0-b64c-71119a497445" volumeName="kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340511    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f836f8b-5b38-4697-b577-46cc16c02f12" volumeName="kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340525    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a575b2-ee3c-4647-b1a6-8b29faed0a41" volumeName="kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340538    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a575b2-ee3c-4647-b1a6-8b29faed0a41" volumeName="kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340550    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27a575b2-ee3c-4647-b1a6-8b29faed0a41" volumeName="kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340562    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="903d010c-458d-4c3a-8c94-8bf685cb6ef3" volumeName="kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca" seLinuxMountContext=""
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.338982    4515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:36 +0000 UTC" deadline="2028-05-20 09:16:35.760335664 +0000 UTC"
Nov 02 07:24:15.340568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340582    4515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13561h52m20.419759364s"
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340574    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d98a9be-246b-45a0-b64c-71119a497445" volumeName="kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340601    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb95b14-6d7c-4241-b22a-dd67fa387dd3" volumeName="kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340614    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cae88a-5ad3-4932-b27e-1a012e018d55" volumeName="kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340627    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf78c693-2031-45f1-910a-c89b469c8b69" volumeName="kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340638    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf78c693-2031-45f1-910a-c89b469c8b69" volumeName="kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340650    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d98a9be-246b-45a0-b64c-71119a497445" volumeName="kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340661    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71b02e54-cab8-41d7-909c-7f0cbe5eab71" volumeName="kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340672    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb95b14-6d7c-4241-b22a-dd67fa387dd3" volumeName="kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340684    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e73491b7-617f-4eb1-b094-69a5db32b449" volumeName="kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340698    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e73491b7-617f-4eb1-b094-69a5db32b449" volumeName="kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340716    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f836f8b-5b38-4697-b577-46cc16c02f12" volumeName="kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340729    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1abc9-0c41-4337-b497-b4ddff25663d" volumeName="kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340741    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53590114-7b6a-4223-8361-f82630bb6d5d" volumeName="kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340753    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53590114-7b6a-4223-8361-f82630bb6d5d" volumeName="kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340779    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3daa4be-adf6-4e9f-8307-98e3c0d180ec" volumeName="kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340793    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf78c693-2031-45f1-910a-c89b469c8b69" volumeName="kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340805    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1abc9-0c41-4337-b497-b4ddff25663d" volumeName="kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340816    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a44cf5-6890-4936-861f-7f26de059789" volumeName="kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340829    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d98a9be-246b-45a0-b64c-71119a497445" volumeName="kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340841    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71b02e54-cab8-41d7-909c-7f0cbe5eab71" volumeName="kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340854    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc523525-bb72-4547-a2dc-c77957b278b0" volumeName="kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340867    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e73491b7-617f-4eb1-b094-69a5db32b449" volumeName="kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340876    4515 factory.go:55] Registering systemd factory
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340880    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2780dc0b-f641-45c0-86f8-769a72bf589c" volumeName="kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340892    4515 factory.go:223] Registration of the systemd container factory successfully
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340893    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1abc9-0c41-4337-b497-b4ddff25663d" volumeName="kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340907    4515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="522f311c-5aeb-455c-b170-d008487dabce" volumeName="kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj" seLinuxMountContext=""
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340933    4515 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 07:24:15.341553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.340941    4515 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.341911    4515 factory.go:153] Registering CRI-O factory
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.341927    4515 factory.go:223] Registration of the crio container factory successfully
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.341981    4515 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.341999    4515 factory.go:103] Registering Raw factory
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.342014    4515 manager.go:1199] Started watching for new ooms in manager
Nov 02 07:24:15.342838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.342341    4515 manager.go:319] Starting recovery of all containers
Nov 02 07:24:15.363768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.363707    4515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 07:24:15.365144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.365122    4515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 07:24:15.365144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.365145    4515 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 07:24:15.365350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.365165    4515 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 07:24:15.365350 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.365207    4515 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 07:24:15.386709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.386675    4515 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 07:24:15.386994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.386969    4515 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:15.388066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.388043    4515 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:15.399010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.398981    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="f4367a2bbbc46bbb6b861a5977ee4c322fc819828753f2e9d3a227c5ef23dbfc" exitCode=0
Nov 02 07:24:15.399010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.399009    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="1026a7ef615fddfeb0668d4810069708937c79be96738f9b29703345bc813eaa" exitCode=0
Nov 02 07:24:15.399010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.399015    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="da2cc743d7a0e57dd6b905707a70218a8f3d3aa303e7d112ff4884f44bacbf0f" exitCode=0
Nov 02 07:24:15.399193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.399025    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="661d28dd7c6e9409f0b2e19866c2aaa158a2ed1e6001716206d8c7c25c843073" exitCode=0
Nov 02 07:24:15.399193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.399034    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="f66ba86d5d6a30be75a53200f63c0525c2c39906edd11e1b702b371f1d6f2fa6" exitCode=0
Nov 02 07:24:15.399193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.399042    4515 generic.go:356] "Generic (PLEG): container finished" podID="27a575b2-ee3c-4647-b1a6-8b29faed0a41" containerID="a61a66be103c06b14343b035b2b65f76f6d163b3c7f1ebbe3a5e813c3d783d72" exitCode=0
Nov 02 07:24:15.401036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.401012    4515 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:15.401180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.401162    4515 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:15.405497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.405475    4515 generic.go:356] "Generic (PLEG): container finished" podID="b271a57358807e5a4d178ecbda3280fd" containerID="1a5a0a5fb24d88372ba4e841447d61150b771da5001791c42581b8bb58c38668" exitCode=0
Nov 02 07:24:15.412792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.412775    4515 manager.go:324] Recovery completed
Nov 02 07:24:15.429421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429399    4515 cpu_manager.go:226] "Starting" policy="none"
Nov 02 07:24:15.429421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429413    4515 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429430    4515 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429541    4515 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429551    4515 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429567    4515 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429572    4515 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429577    4515 policy_none.go:50] "Start"
Nov 02 07:24:15.429586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429585    4515 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 07:24:15.429796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.429594    4515 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 07:24:15.433089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.433050    4515 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 07:24:15.433089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.433064    4515 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 07:24:15.433089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.433070    4515 policy_none.go:44] "Start"
Nov 02 07:24:15.437126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.437097    4515 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 07:24:15.439616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.439603    4515 manager.go:341] "Starting Device Plugin manager"
Nov 02 07:24:15.440189 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.439627    4515 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 07:24:15.440228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440193    4515 server.go:87] "Starting device plugin registration server"
Nov 02 07:24:15.440468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440456    4515 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 07:24:15.440505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440470    4515 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 07:24:15.440617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440595    4515 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 07:24:15.440733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440697    4515 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 07:24:15.440796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440743    4515 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 07:24:15.440796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440749    4515 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 07:24:15.440899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.440802    4515 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:24:15.440732Z","UUID":"58499567-57a6-413a-95f2-9823ee64233b","Handler":null,"Name":"","Endpoint":""}
Nov 02 07:24:15.442113 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.442095    4515 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 07:24:15.442203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.442100    4515 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 07:24:15.442203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.442181    4515 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 07:24:15.465325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465260    4515 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"]
Nov 02 07:24:15.465937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465884    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal" event={"ID":"735495427ccc6de5684bdc4a015d248e","Type":"ContainerStarted","Data":"bf04a3b56358c5ef4f2d8c9d90084db2d0148080d4dab07a51eadd6dec0401a9"}
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465950    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal" event={"ID":"735495427ccc6de5684bdc4a015d248e","Type":"ContainerStarted","Data":"e57453c917c9dc0a18a9cf5b0b1bacc73636f0ffd34a9883a2b88adde411fe17"}
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465973    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerStarted","Data":"65e30be68534007df3d5bada78ebb00ba15f4806ead9edcc207b74bc58c0803e"}
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465989    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerDied","Data":"1a5a0a5fb24d88372ba4e841447d61150b771da5001791c42581b8bb58c38668"}
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.466001    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal" event={"ID":"b271a57358807e5a4d178ecbda3280fd","Type":"ContainerStarted","Data":"79736bfdfd338e91e622e8162a15220468bf784e2fb69dad3b8cd2d003293ae3"}
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.466000    4515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.466039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.465956    4515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.489280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.489253    4515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:15.489419 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.489331    4515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.491119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.491102    4515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:15.491198 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:15.491140    4515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.540800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.540766    4515 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:24:15.542337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.542300    4515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:24:15.542458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.542353    4515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:24:15.542458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.542364    4515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-156.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:24:15.542458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.542401    4515 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.577515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.577488    4515 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.577647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.577556    4515 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.588320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.588281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.588320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.588316    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.588457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.588334    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.688946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688864    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.688946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688899    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.688946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688919    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.689175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688967    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/735495427ccc6de5684bdc4a015d248e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" (UID: \"735495427ccc6de5684bdc4a015d248e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.689175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688988    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:15.689175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:15.688990    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b271a57358807e5a4d178ecbda3280fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" (UID: \"b271a57358807e5a4d178ecbda3280fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:16.324706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.324664    4515 apiserver.go:52] "Watching apiserver"
Nov 02 07:24:16.367211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.367171    4515 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 07:24:16.367680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.367654    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal","openshift-network-diagnostics/network-check-target-xq8m6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l","openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4","openshift-dns/node-resolver-97v78","openshift-insights/periodic-gathering-8qx5v-wgjd6","openshift-dns/dns-default-2rzs7","openshift-network-operator/iptables-alerter-vnz9b","kube-system/global-pull-secret-syncer-brmn4","openshift-insights/insights-runtime-extractor-kc2t9","openshift-multus/network-metrics-daemon-qnkkt","openshift-multus/multus-wxks2","openshift-service-ca/service-ca-5fffc97bdf-wwl6t","openshift-ovn-kubernetes/ovnkube-node-cljfx","openshift-multus/multus-additional-cni-plugins-6bt4h","kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal","openshift-image-registry/node-ca-mldbl","kube-system/konnectivity-agent-qfghm","openshift-cluster-node-tuning-operator/tuned-cwm7p","openshift-ingress-canary/ingress-canary-9dfxs"]
Nov 02 07:24:16.373952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.373922    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.374668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.374648    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.375058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.375037    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.383259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.383236    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.383457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.383438    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.383712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.383695    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 07:24:16.383797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.383713    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 07:24:16.384107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384087    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.384160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384117    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 07:24:16.384160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384117    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.384232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384163    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\""
Nov 02 07:24:16.384232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384182    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 07:24:16.384232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384202    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.384232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384230    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 07:24:16.384377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384273    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\""
Nov 02 07:24:16.384377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.384303    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\""
Nov 02 07:24:16.394728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.394710    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.394779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.394762    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 07:24:16.395341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.395301    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 07:24:16.395431 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.395304    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\""
Nov 02 07:24:16.395526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.395513    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.396076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396057    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\""
Nov 02 07:24:16.396141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396058    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-977zc\""
Nov 02 07:24:16.396141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396087    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\""
Nov 02 07:24:16.396141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396063    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\""
Nov 02 07:24:16.396300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396275    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 07:24:16.396434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396355    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.396434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396371    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 07:24:16.396434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396389    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 07:24:16.396434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396402    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 07:24:16.396616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396440    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 07:24:16.396616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396512    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.396616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396535    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.396616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396542    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.396616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396566    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 07:24:16.396796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396783    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396828    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396830    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396848    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396895    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-48668\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396897    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 07:24:16.396969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.396957    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397010    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397014    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397053    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397093    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397130    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397154    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397165    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397190    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397195    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 07:24:16.397287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397220    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.397652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397295    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\""
Nov 02 07:24:16.397652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397464    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 07:24:16.397787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397773    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\""
Nov 02 07:24:16.397887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.397871    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ztvhn\""
Nov 02 07:24:16.408209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408188    4515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:16.408354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408320    4515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408370    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408382    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408372    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408419    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408423    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408448    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.408471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408451    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.408750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408568    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\""
Nov 02 07:24:16.408970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408954    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.409029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408996    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 07:24:16.409029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.408997    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 07:24:16.409029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.409025    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 07:24:16.409550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.409537    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 07:24:16.431440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.431417    4515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:16.431541 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:16.431462    4515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:16.432000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.431986    4515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:16.432054 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:16.432017    4515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-156.ec2.internal"
Nov 02 07:24:16.440092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.440067    4515 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 07:24:16.493079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493043    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.493079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493079    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.493303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493100    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.493303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493172    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.493303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493205    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jnp\" (UniqueName: \"kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.493303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493226    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.493303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493245    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.493512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493421    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.493512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493432    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.493512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493461    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.493512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493484    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493519    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493548    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgsr\" (UniqueName: \"kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493609    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493623    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-original-pull-secret\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493634    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b02e54-cab8-41d7-909c-7f0cbe5eab71-tmp-dir\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493640    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.493686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493669    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6js\" (UniqueName: \"kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493701    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85d7r\" (UniqueName: \"kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493723    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493745    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493768    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493788    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493815    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493842    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493920    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.493997    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7zj\" (UniqueName: \"kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.494026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494022    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-script-lib\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494084    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf78c693-2031-45f1-910a-c89b469c8b69-signing-cabundle\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494086    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494136    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494191    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494235    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494254    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494270    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494292    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494338    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494398    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-tuned\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494409    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.494504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494490    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndwz\" (UniqueName: \"kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494519    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494566    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494590    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494585    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-ovnkube-config\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494675    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494715    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh2r\" (UniqueName: \"kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494740    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494755    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e73491b7-617f-4eb1-b094-69a5db32b449-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494786    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d98a9be-246b-45a0-b64c-71119a497445-env-overrides\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494839    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494892    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494911    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494926    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.494987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.494944    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495009    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-tmp\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495045    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495071    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495094    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495118    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495162    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495179    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495194    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495243    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495272    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495303    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495353    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f836f8b-5b38-4697-b577-46cc16c02f12-konnectivity-ca\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495418    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88cae88a-5ad3-4932-b27e-1a012e018d55-config-volume\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.495475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495419    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghdw\" (UniqueName: \"kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495500    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/903d010c-458d-4c3a-8c94-8bf685cb6ef3-serviceca\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495516    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbm9\" (UniqueName: \"kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495543    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsnh5\" (UniqueName: \"kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495560    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495576    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495617    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495643    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-daemon-config\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495690    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495745    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495789    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495820    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495876    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495923    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495961    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.495996    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496027    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk79w\" (UniqueName: \"kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.496043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496044    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6km\" (UniqueName: \"kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496085    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496203    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496234    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496294    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496351    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496378    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf78c693-2031-45f1-910a-c89b469c8b69-signing-key\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496402    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496443    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496488    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496533    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496543    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9cw\" (UniqueName: \"kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496558    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496582    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.496648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496608    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496712    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496754    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496757    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496810    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496835    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cae88a-5ad3-4932-b27e-1a012e018d55-tmp-dir\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496847    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ec1abc9-0c41-4337-b497-b4ddff25663d-cni-binary-copy\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496870    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496891    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496925    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496929    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e73491b7-617f-4eb1-b094-69a5db32b449-data-volume\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.496983    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497008    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.497113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497027    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497136    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497160    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497169    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497179    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75sj\" (UniqueName: \"kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj\") pod \"migrator-66df5d496b-9jvf4\" (UID: \"522f311c-5aeb-455c-b170-d008487dabce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497198    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497220    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497338    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497382    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497401    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2780dc0b-f641-45c0-86f8-769a72bf589c-iptables-alerter-script\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497495    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497594    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f836f8b-5b38-4697-b577-46cc16c02f12-agent-certs\") pod \"konnectivity-agent-qfghm\" (UID: \"0f836f8b-5b38-4697-b577-46cc16c02f12\") " pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497618    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d98a9be-246b-45a0-b64c-71119a497445-ovn-node-metrics-cert\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.497685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.497664    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97fa6161-e719-4d4b-8d83-5188300c9a4d-cert\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:16.516072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.516042    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85d7r\" (UniqueName: \"kubernetes.io/projected/903d010c-458d-4c3a-8c94-8bf685cb6ef3-kube-api-access-85d7r\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.516241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.516193    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7zj\" (UniqueName: \"kubernetes.io/projected/3ec1abc9-0c41-4337-b497-b4ddff25663d-kube-api-access-qd7zj\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.516241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.516206    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgsr\" (UniqueName: \"kubernetes.io/projected/27a575b2-ee3c-4647-b1a6-8b29faed0a41-kube-api-access-nvgsr\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.516613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.516597    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jnp\" (UniqueName: \"kubernetes.io/projected/e73491b7-617f-4eb1-b094-69a5db32b449-kube-api-access-g2jnp\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.516795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.516776    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndwz\" (UniqueName: \"kubernetes.io/projected/88cae88a-5ad3-4932-b27e-1a012e018d55-kube-api-access-7ndwz\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.521087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.521069    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6js\" (UniqueName: \"kubernetes.io/projected/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-kube-api-access-dz6js\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.521162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.521117    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh2r\" (UniqueName: \"kubernetes.io/projected/97fa6161-e719-4d4b-8d83-5188300c9a4d-kube-api-access-xnh2r\") pod \"ingress-canary-9dfxs\" (UID: \"97fa6161-e719-4d4b-8d83-5188300c9a4d\") " pod="openshift-ingress-canary/ingress-canary-9dfxs"
Nov 02 07:24:16.523907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.523891    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghdw\" (UniqueName: \"kubernetes.io/projected/43a44cf5-6890-4936-861f-7f26de059789-kube-api-access-5ghdw\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.526299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526122    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9cw\" (UniqueName: \"kubernetes.io/projected/d7a903fa-2c1d-47ed-8701-1c220c920b57-kube-api-access-hs9cw\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.526299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526233    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6km\" (UniqueName: \"kubernetes.io/projected/4d98a9be-246b-45a0-b64c-71119a497445-kube-api-access-cl6km\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.526299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526292    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbm9\" (UniqueName: \"kubernetes.io/projected/71b02e54-cab8-41d7-909c-7f0cbe5eab71-kube-api-access-2lbm9\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.526299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526320    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"periodic-gathering-8qx5v-wgjd6\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") " pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:24:16.526529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsnh5\" (UniqueName: \"kubernetes.io/projected/2780dc0b-f641-45c0-86f8-769a72bf589c-kube-api-access-hsnh5\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.526529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526375    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dlg9\" (UniqueName: \"kubernetes.io/projected/dc523525-bb72-4547-a2dc-c77957b278b0-kube-api-access-4dlg9\") pod \"network-check-target-xq8m6\" (UID: \"dc523525-bb72-4547-a2dc-c77957b278b0\") " pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:16.526529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526449    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75sj\" (UniqueName: \"kubernetes.io/projected/522f311c-5aeb-455c-b170-d008487dabce-kube-api-access-p75sj\") pod \"migrator-66df5d496b-9jvf4\" (UID: \"522f311c-5aeb-455c-b170-d008487dabce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9jvf4"
Nov 02 07:24:16.526639 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.526624    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk79w\" (UniqueName: \"kubernetes.io/projected/bf78c693-2031-45f1-910a-c89b469c8b69-kube-api-access-lk79w\") pod \"service-ca-5fffc97bdf-wwl6t\" (UID: \"bf78c693-2031-45f1-910a-c89b469c8b69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wwl6t"
Nov 02 07:24:16.598297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598205    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.598297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598251    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.598297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598275    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.598297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598296    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598342    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598368    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598384    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2780dc0b-f641-45c0-86f8-769a72bf589c-host-slash\") pod \"iptables-alerter-vnz9b\" (UID: \"2780dc0b-f641-45c0-86f8-769a72bf589c\") " pod="openshift-network-operator/iptables-alerter-vnz9b"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598383    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-system-cni-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598394    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598410    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-multus-certs\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598426    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-etc-selinux\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598440    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598428    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-netns\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598478    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598481    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-conf-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598507    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598519    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71b02e54-cab8-41d7-909c-7f0cbe5eab71-hosts-file\") pod \"node-resolver-97v78\" (UID: \"71b02e54-cab8-41d7-909c-7f0cbe5eab71\") " pod="openshift-dns/node-resolver-97v78"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598520    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598548    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-cnibin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598549    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598580    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598580    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-hostroot\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598609    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-slash\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598626    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.598629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598646    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598671    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598689    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-netns\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598700    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598712    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-ovn\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598693    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-etc-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598726    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598737    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-var-lib-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598751    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598773    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598788    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598805    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598797    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-modprobe-d\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598833    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598834    4515 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598849    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598859    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-sys-fs\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598859    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598850    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-run-k8s-cni-cncf-io\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598889    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-netd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598897    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-bin\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598898    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-systemd\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598921    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598939    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.598994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-host\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599014    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599032    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599058    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-etc-kubernetes\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599086    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599140    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-cnibin\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599143    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599140    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-kubelet-config\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599191    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:16.599189    4515 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599222    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599232    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-socket-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599256    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:16.599284    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls podName:e73491b7-617f-4eb1-b094-69a5db32b449 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:17.099264382 +0000 UTC m=+1.861701286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kc2t9" (UID: "e73491b7-617f-4eb1-b094-69a5db32b449") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599327    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-multus-socket-dir-parent\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599344    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-lib-modules\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599329    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599383    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599383    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysctl-conf\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599384    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599413    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-systemd-units\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599454    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-kubelet\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599462    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599506    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599532    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599552    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-openvswitch\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599556    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599557    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-cni-bin\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599592    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-kubelet\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.599561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599586    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-sysconfig\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599594    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599637    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-registration-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599642    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599669    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599723    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599732    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-run\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599733    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-run-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599748    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599756    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-host-var-lib-cni-multus\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599775    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e73491b7-617f-4eb1-b094-69a5db32b449-crio-socket\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599778    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599802    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599831    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27a575b2-ee3c-4647-b1a6-8b29faed0a41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bt4h\" (UID: \"27a575b2-ee3c-4647-b1a6-8b29faed0a41\") " pod="openshift-multus/multus-additional-cni-plugins-6bt4h"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599833    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-etc-kubernetes\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599861    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599872    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-log-socket\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599890    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599907    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599915    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599888    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-sys\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599964    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43a44cf5-6890-4936-861f-7f26de059789-device-dir\") pod \"aws-ebs-csi-driver-node-brb7l\" (UID: \"43a44cf5-6890-4936-861f-7f26de059789\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brb7l"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599965    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb95b14-6d7c-4241-b22a-dd67fa387dd3-var-lib-kubelet\") pod \"tuned-cwm7p\" (UID: \"7cb95b14-6d7c-4241-b22a-dd67fa387dd3\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm7p"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.599982    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600005    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600029    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600056    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600059    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-run-systemd\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600092    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600101    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d98a9be-246b-45a0-b64c-71119a497445-node-log\") pod \"ovnkube-node-cljfx\" (UID: \"4d98a9be-246b-45a0-b64c-71119a497445\") " pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600105    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/903d010c-458d-4c3a-8c94-8bf685cb6ef3-host\") pod \"node-ca-mldbl\" (UID: \"903d010c-458d-4c3a-8c94-8bf685cb6ef3\") " pod="openshift-image-registry/node-ca-mldbl"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600119    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600145    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3daa4be-adf6-4e9f-8307-98e3c0d180ec-dbus\") pod \"global-pull-secret-syncer-brmn4\" (UID: \"a3daa4be-adf6-4e9f-8307-98e3c0d180ec\") " pod="kube-system/global-pull-secret-syncer-brmn4"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.600160    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ec1abc9-0c41-4337-b497-b4ddff25663d-system-cni-dir\") pod \"multus-wxks2\" (UID: \"3ec1abc9-0c41-4337-b497-b4ddff25663d\") " pod="openshift-multus/multus-wxks2"
Nov 02 07:24:16.601537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.601298    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a903fa-2c1d-47ed-8701-1c220c920b57-metrics-certs\") pod \"network-metrics-daemon-qnkkt\" (UID: \"d7a903fa-2c1d-47ed-8701-1c220c920b57\") " pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.602631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.602489    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88cae88a-5ad3-4932-b27e-1a012e018d55-metrics-tls\") pod \"dns-default-2rzs7\" (UID: \"88cae88a-5ad3-4932-b27e-1a012e018d55\") " pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.674223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.674189    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnkkt"
Nov 02 07:24:16.675388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.675370    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:16.999337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:16.999302    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnkkt"]
Nov 02 07:24:17.000927 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:17.000898    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a903fa_2c1d_47ed_8701_1c220c920b57.slice/crio-1f990a0c32bc4e4537a5003736bae4889688d598da3d9ee78f0652966dca8c24 WatchSource:0}: Error finding container 1f990a0c32bc4e4537a5003736bae4889688d598da3d9ee78f0652966dca8c24: Status 404 returned error can't find the container with id 1f990a0c32bc4e4537a5003736bae4889688d598da3d9ee78f0652966dca8c24
Nov 02 07:24:17.002923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.002907    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:24:17.036423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.036399    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rzs7"]
Nov 02 07:24:17.037670 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:17.037646    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cae88a_5ad3_4932_b27e_1a012e018d55.slice/crio-4b937ac0a0af7be8905adc771033014cebf50b3063f328541d01e30ff84c8b98 WatchSource:0}: Error finding container 4b937ac0a0af7be8905adc771033014cebf50b3063f328541d01e30ff84c8b98: Status 404 returned error can't find the container with id 4b937ac0a0af7be8905adc771033014cebf50b3063f328541d01e30ff84c8b98
Nov 02 07:24:17.102443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.102412    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:17.104614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.104590    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e73491b7-617f-4eb1-b094-69a5db32b449-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc2t9\" (UID: \"e73491b7-617f-4eb1-b094-69a5db32b449\") " pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:17.275506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.275475    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc2t9"
Nov 02 07:24:17.411529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.411494    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rzs7" event={"ID":"88cae88a-5ad3-4932-b27e-1a012e018d55","Type":"ContainerStarted","Data":"4b937ac0a0af7be8905adc771033014cebf50b3063f328541d01e30ff84c8b98"}
Nov 02 07:24:17.412675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.412647    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnkkt" event={"ID":"d7a903fa-2c1d-47ed-8701-1c220c920b57","Type":"ContainerStarted","Data":"1f990a0c32bc4e4537a5003736bae4889688d598da3d9ee78f0652966dca8c24"}
Nov 02 07:24:17.601071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:17.601030    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc2t9"]
Nov 02 07:24:17.604029 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:17.603993    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73491b7_617f_4eb1_b094_69a5db32b449.slice/crio-85f0aa731bca7c130a686d09ffbbdbac0f66321d24af6882e707ada99870b304 WatchSource:0}: Error finding container 85f0aa731bca7c130a686d09ffbbdbac0f66321d24af6882e707ada99870b304: Status 404 returned error can't find the container with id 85f0aa731bca7c130a686d09ffbbdbac0f66321d24af6882e707ada99870b304
Nov 02 07:24:18.419094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:18.419063    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnkkt" event={"ID":"d7a903fa-2c1d-47ed-8701-1c220c920b57","Type":"ContainerStarted","Data":"9b204423ce2420ae61d2624a380d215a98c6e4a8e40b05f5c7a65d556a5e2bd4"}
Nov 02 07:24:18.421087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:18.421063    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc2t9" event={"ID":"e73491b7-617f-4eb1-b094-69a5db32b449","Type":"ContainerStarted","Data":"228a60e3b892e38393ea1df5605693ed9f7137cda85818920429f47b1107612f"}
Nov 02 07:24:18.421158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:18.421110    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc2t9" event={"ID":"e73491b7-617f-4eb1-b094-69a5db32b449","Type":"ContainerStarted","Data":"85f0aa731bca7c130a686d09ffbbdbac0f66321d24af6882e707ada99870b304"}
Nov 02 07:24:19.147189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.147116    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qfghm"
Nov 02 07:24:19.426923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.426829    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc2t9" event={"ID":"e73491b7-617f-4eb1-b094-69a5db32b449","Type":"ContainerStarted","Data":"6f062f323c31afe636d73c2503b1725f515fddc7866f8a42c91a038015deb1e1"}
Nov 02 07:24:19.428849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.428814    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rzs7" event={"ID":"88cae88a-5ad3-4932-b27e-1a012e018d55","Type":"ContainerStarted","Data":"0236edef4cf75b19d246e82bd1db8df31b2a5920da412392b6f79e5835ff21be"}
Nov 02 07:24:19.428966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.428862    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rzs7" event={"ID":"88cae88a-5ad3-4932-b27e-1a012e018d55","Type":"ContainerStarted","Data":"57c8b19ebef9d2d93cf1569f96ab1ec7b595b303bc3a8f8d0e9e03a2f9f150ed"}
Nov 02 07:24:19.428966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.428905    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:19.430805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:19.430766    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnkkt" event={"ID":"d7a903fa-2c1d-47ed-8701-1c220c920b57","Type":"ContainerStarted","Data":"3930eb927982d0507c67ed5b54e37f189ae27365d2e4e2325a2bb9e628af1308"}
Nov 02 07:24:20.303489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:20.303457    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:20.436177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:20.436129    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc2t9" event={"ID":"e73491b7-617f-4eb1-b094-69a5db32b449","Type":"ContainerStarted","Data":"bf90e4ae7b30e14ad150b4caec8dbd95b57e563e97a248aba1901e4ee58006f1"}
Nov 02 07:24:21.108459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:21.108432    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:23.586065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:23.586036    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:23.586455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:23.586197    4515 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 07:24:26.272911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.272874    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"]
Nov 02 07:24:26.275813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.275790    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:26.284421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.284396    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-l9ngq"]
Nov 02 07:24:26.287066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.287047    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.297442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.297414    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zd2q8\""
Nov 02 07:24:26.297654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.297490    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 07:24:26.298154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.298132    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"]
Nov 02 07:24:26.309418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.309393    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 07:24:26.309534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.309452    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 07:24:26.309951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.309932    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l9ngq"]
Nov 02 07:24:26.359164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.359133    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56f58c559d-klv5k"]
Nov 02 07:24:26.362054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.362037    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.363126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.363105    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02f3384d-e450-4445-929f-ea4b686859d9-tmp\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.363208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.363148    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0673a2b-eb6a-4d22-af35-c12eaf913948-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vnmcv\" (UID: \"d0673a2b-eb6a-4d22-af35-c12eaf913948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:26.363208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.363183    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pg4\" (UniqueName: \"kubernetes.io/projected/02f3384d-e450-4445-929f-ea4b686859d9-kube-api-access-l5pg4\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.397759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.397734    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 07:24:26.398394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.398375    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f58c559d-klv5k"]
Nov 02 07:24:26.400130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.400111    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 07:24:26.400130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.400126    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 07:24:26.400279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.400117    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-c945l\""
Nov 02 07:24:26.400279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.400165    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 07:24:26.400279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.400235    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4hqr\""
Nov 02 07:24:26.463760 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.463720    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-certificates\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.463916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.463767    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0673a2b-eb6a-4d22-af35-c12eaf913948-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vnmcv\" (UID: \"d0673a2b-eb6a-4d22-af35-c12eaf913948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:26.463916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.463831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pg4\" (UniqueName: \"kubernetes.io/projected/02f3384d-e450-4445-929f-ea4b686859d9-kube-api-access-l5pg4\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.464019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.463953    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6r5\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-kube-api-access-tg6r5\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.463982    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4ab6de3-44f6-455d-8345-8eda5144cdce-ca-trust-extracted\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464012    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-image-registry-private-configuration\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464097    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02f3384d-e450-4445-929f-ea4b686859d9-tmp\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.464233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464195    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-tls\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464342    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-trusted-ca\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464373    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-installation-pull-secrets\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464389    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-bound-sa-token\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.464542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.464504    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02f3384d-e450-4445-929f-ea4b686859d9-tmp\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.466201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.466182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0673a2b-eb6a-4d22-af35-c12eaf913948-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vnmcv\" (UID: \"d0673a2b-eb6a-4d22-af35-c12eaf913948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:26.487472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.487439    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pg4\" (UniqueName: \"kubernetes.io/projected/02f3384d-e450-4445-929f-ea4b686859d9-kube-api-access-l5pg4\") pod \"downloads-6dd789c8b7-l9ngq\" (UID: \"02f3384d-e450-4445-929f-ea4b686859d9\") " pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.565641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565552    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-trusted-ca\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565593    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-installation-pull-secrets\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565611    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-bound-sa-token\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565749    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-certificates\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565799    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6r5\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-kube-api-access-tg6r5\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565826    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4ab6de3-44f6-455d-8345-8eda5144cdce-ca-trust-extracted\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565863    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-image-registry-private-configuration\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.565918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.565898    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-tls\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.566302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.566268    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4ab6de3-44f6-455d-8345-8eda5144cdce-ca-trust-extracted\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.566632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.566610    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-certificates\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.566903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.566878    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab6de3-44f6-455d-8345-8eda5144cdce-trusted-ca\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.568097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.568073    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-installation-pull-secrets\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.568193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.568118    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4ab6de3-44f6-455d-8345-8eda5144cdce-image-registry-private-configuration\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.568321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.568290    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-registry-tls\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.586778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.586751    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:26.591211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.591129    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6r5\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-kube-api-access-tg6r5\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.591302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.591278    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4ab6de3-44f6-455d-8345-8eda5144cdce-bound-sa-token\") pod \"image-registry-56f58c559d-klv5k\" (UID: \"e4ab6de3-44f6-455d-8345-8eda5144cdce\") " pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.597426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.597407    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:26.672803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.672620    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:26.886811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.886745    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"]
Nov 02 07:24:26.889860 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:26.889834    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0673a2b_eb6a_4d22_af35_c12eaf913948.slice/crio-9dc3c456ce9a87f42afa5eb7d8c9e05d96a358de490a222bb7310db7cf1e4a5a WatchSource:0}: Error finding container 9dc3c456ce9a87f42afa5eb7d8c9e05d96a358de490a222bb7310db7cf1e4a5a: Status 404 returned error can't find the container with id 9dc3c456ce9a87f42afa5eb7d8c9e05d96a358de490a222bb7310db7cf1e4a5a
Nov 02 07:24:26.914844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.914821    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l9ngq"]
Nov 02 07:24:26.916152 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:26.916127    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f3384d_e450_4445_929f_ea4b686859d9.slice/crio-475523da67b580bdf43663a63956d2f4389fe2686f4d25d5fbc4e6e739c26621 WatchSource:0}: Error finding container 475523da67b580bdf43663a63956d2f4389fe2686f4d25d5fbc4e6e739c26621: Status 404 returned error can't find the container with id 475523da67b580bdf43663a63956d2f4389fe2686f4d25d5fbc4e6e739c26621
Nov 02 07:24:26.969770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:26.969701    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f58c559d-klv5k"]
Nov 02 07:24:26.971644 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:26.971617    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ab6de3_44f6_455d_8345_8eda5144cdce.slice/crio-341e461f2fb29250e680d0f3915de5e2c584e8b16ffa928d1058f4652905322b WatchSource:0}: Error finding container 341e461f2fb29250e680d0f3915de5e2c584e8b16ffa928d1058f4652905322b: Status 404 returned error can't find the container with id 341e461f2fb29250e680d0f3915de5e2c584e8b16ffa928d1058f4652905322b
Nov 02 07:24:27.461369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.461332    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f58c559d-klv5k" event={"ID":"e4ab6de3-44f6-455d-8345-8eda5144cdce","Type":"ContainerStarted","Data":"133848ae00dd89d261c938dc09ec9ea930eb52ca655116f3b1eed4ee6094b0b1"}
Nov 02 07:24:27.461841 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.461398    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f58c559d-klv5k" event={"ID":"e4ab6de3-44f6-455d-8345-8eda5144cdce","Type":"ContainerStarted","Data":"341e461f2fb29250e680d0f3915de5e2c584e8b16ffa928d1058f4652905322b"}
Nov 02 07:24:27.461841 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.461445    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:27.463332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.463287    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv" event={"ID":"d0673a2b-eb6a-4d22-af35-c12eaf913948","Type":"ContainerStarted","Data":"9dc3c456ce9a87f42afa5eb7d8c9e05d96a358de490a222bb7310db7cf1e4a5a"}
Nov 02 07:24:27.465549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.465519    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l9ngq" event={"ID":"02f3384d-e450-4445-929f-ea4b686859d9","Type":"ContainerStarted","Data":"475523da67b580bdf43663a63956d2f4389fe2686f4d25d5fbc4e6e739c26621"}
Nov 02 07:24:27.509229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:27.509163    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56f58c559d-klv5k" podStartSLOduration=1.509144507 podStartE2EDuration="1.509144507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:27.50871698 +0000 UTC m=+12.271153912" watchObservedRunningTime="2026-11-02 07:24:27.509144507 +0000 UTC m=+12.271581414"
Nov 02 07:24:28.471100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:28.471058    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv" event={"ID":"d0673a2b-eb6a-4d22-af35-c12eaf913948","Type":"ContainerStarted","Data":"98752948e698026e35d8ac79dfbcccff47e4d09d47e7eed136bdede9559571f7"}
Nov 02 07:24:28.511133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:28.511070    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv" podStartSLOduration=1.406666126 podStartE2EDuration="2.511051248s" totalImagesPullingTime="1.104385122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:28.510765558 +0000 UTC m=+13.273202463" watchObservedRunningTime="2026-11-02 07:24:28.511051248 +0000 UTC m=+13.273488150"
Nov 02 07:24:28.997944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:28.997913    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:24:29.001039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.001015    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.031414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.031388    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 07:24:29.031414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.031405    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 07:24:29.031610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.031406    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 07:24:29.031610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.031559    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jcgqg\""
Nov 02 07:24:29.032484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.032459    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:24:29.035668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.035647    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 07:24:29.035799 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.035678    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 07:24:29.092672 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.092842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092687    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.092842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092714    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.092842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092817    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.092842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.093005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092861    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.093005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.092902    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsst8\" (UniqueName: \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193584    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193624    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193729    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193780    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193822    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.193875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193867    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsst8\" (UniqueName: \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.194064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.193909    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.194618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.194443    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.194618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.194522    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.194618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.194557    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.195018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.194998    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.196489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.196467    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.196628 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.196608    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.218201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.218175    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsst8\" (UniqueName: \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\") pod \"console-55ff76bb47-f5v8j\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") " pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.313626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.313579    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:29.439226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.439195    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2rzs7"
Nov 02 07:24:29.474925 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.474893    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:29.480544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.480514    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vnmcv"
Nov 02 07:24:29.619934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:29.619909    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:24:29.622080 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:29.622050    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394adac3_7bac_448a_b05d_858af9a0b9f2.slice/crio-6ec8ae4ed5314d96a52179d4926adac112df613fc3f360cc3f3e91409d6f29d3 WatchSource:0}: Error finding container 6ec8ae4ed5314d96a52179d4926adac112df613fc3f360cc3f3e91409d6f29d3: Status 404 returned error can't find the container with id 6ec8ae4ed5314d96a52179d4926adac112df613fc3f360cc3f3e91409d6f29d3
Nov 02 07:24:30.480517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.480476    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff76bb47-f5v8j" event={"ID":"394adac3-7bac-448a-b05d-858af9a0b9f2","Type":"ContainerStarted","Data":"6ec8ae4ed5314d96a52179d4926adac112df613fc3f360cc3f3e91409d6f29d3"}
Nov 02 07:24:30.680813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.679782    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"]
Nov 02 07:24:30.683173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.683146    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.698573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.698546    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 07:24:30.698701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.698650    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 07:24:30.698757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.698707    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 07:24:30.711162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.711138    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 07:24:30.711350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.711333    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 07:24:30.711443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.711390    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fshgm\""
Nov 02 07:24:30.712536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.712515    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"]
Nov 02 07:24:30.806982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.806938    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6l8\" (UniqueName: \"kubernetes.io/projected/296b6a48-acbb-4fc0-b41e-f03d344e8505-kube-api-access-dt6l8\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.807164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.807080    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.807164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.807140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/296b6a48-acbb-4fc0-b41e-f03d344e8505-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.807282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.807176    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.908515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.908475    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/296b6a48-acbb-4fc0-b41e-f03d344e8505-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.908694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.908520    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.908694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.908566    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6l8\" (UniqueName: \"kubernetes.io/projected/296b6a48-acbb-4fc0-b41e-f03d344e8505-kube-api-access-dt6l8\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.908694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.908628    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.909677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.909290    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/296b6a48-acbb-4fc0-b41e-f03d344e8505-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.911710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.911685    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.912079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.912058    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/296b6a48-acbb-4fc0-b41e-f03d344e8505-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:30.933603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.933557    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6l8\" (UniqueName: \"kubernetes.io/projected/296b6a48-acbb-4fc0-b41e-f03d344e8505-kube-api-access-dt6l8\") pod \"prometheus-operator-9cd89b74c-wszcp\" (UID: \"296b6a48-acbb-4fc0-b41e-f03d344e8505\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:31.000537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:30.999958    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"
Nov 02 07:24:31.313078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:31.313032    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wszcp"]
Nov 02 07:24:31.317490 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:31.317455    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296b6a48_acbb_4fc0_b41e_f03d344e8505.slice/crio-d3876a935793d373b14b97a0815eb780f5e56cc87087d55eb1aaf00e70703594 WatchSource:0}: Error finding container d3876a935793d373b14b97a0815eb780f5e56cc87087d55eb1aaf00e70703594: Status 404 returned error can't find the container with id d3876a935793d373b14b97a0815eb780f5e56cc87087d55eb1aaf00e70703594
Nov 02 07:24:31.485374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:31.485334    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp" event={"ID":"296b6a48-acbb-4fc0-b41e-f03d344e8505","Type":"ContainerStarted","Data":"d3876a935793d373b14b97a0815eb780f5e56cc87087d55eb1aaf00e70703594"}
Nov 02 07:24:33.496553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:33.496510    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp" event={"ID":"296b6a48-acbb-4fc0-b41e-f03d344e8505","Type":"ContainerStarted","Data":"7922291ca7dc8ed053f3de13fb28468bed109cb1b492293b2e6246beb509cb4c"}
Nov 02 07:24:33.496981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:33.496582    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp" event={"ID":"296b6a48-acbb-4fc0-b41e-f03d344e8505","Type":"ContainerStarted","Data":"9474e3ca17c855bb0fe6129cf330ed6b659c8fee60242cbbb567bb22f64a9297"}
Nov 02 07:24:33.498591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:33.498563    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff76bb47-f5v8j" event={"ID":"394adac3-7bac-448a-b05d-858af9a0b9f2","Type":"ContainerStarted","Data":"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"}
Nov 02 07:24:33.539671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:33.539609    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wszcp" podStartSLOduration=2.050908921 podStartE2EDuration="3.539589441s" totalImagesPullingTime="1.48868052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:33.538465776 +0000 UTC m=+18.300902682" watchObservedRunningTime="2026-11-02 07:24:33.539589441 +0000 UTC m=+18.302026348"
Nov 02 07:24:33.582803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:33.582728    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55ff76bb47-f5v8j" podStartSLOduration=2.507670817 podStartE2EDuration="5.582706667s" totalImagesPullingTime="3.07503585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:33.582411762 +0000 UTC m=+18.344848669" watchObservedRunningTime="2026-11-02 07:24:33.582706667 +0000 UTC m=+18.345143570"
Nov 02 07:24:35.356079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.356044    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rs52m"]
Nov 02 07:24:35.359897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.359870    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.424411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.424338    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 07:24:35.424594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.424446    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 07:24:35.424594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.424494    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zwd5q\""
Nov 02 07:24:35.424594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.424564    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 07:24:35.547833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.547793    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-wtmp\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.547842    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-sys\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.547868    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-tls\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.547931    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.548000    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q98z\" (UniqueName: \"kubernetes.io/projected/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-kube-api-access-6q98z\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.548086    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-textfile\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.548160    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-root\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.548207    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-metrics-client-ca\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.548290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.548272    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649038    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-root\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649099    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-metrics-client-ca\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649137    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649169    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-root\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-wtmp\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649243    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-sys\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649269    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-tls\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649292    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649320    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-wtmp\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649333    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q98z\" (UniqueName: \"kubernetes.io/projected/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-kube-api-access-6q98z\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649677    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-textfile\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649754    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-sys\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.649973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.649951    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-metrics-client-ca\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.650057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.650038    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-textfile\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.650453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.650430    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.652026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.651982    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.652427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.652404    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-node-exporter-tls\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.673291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.673264    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q98z\" (UniqueName: \"kubernetes.io/projected/e5d75c53-9177-4bbb-a6f8-4f5d358083c2-kube-api-access-6q98z\") pod \"node-exporter-rs52m\" (UID: \"e5d75c53-9177-4bbb-a6f8-4f5d358083c2\") " pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.677147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.677124    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rs52m"
Nov 02 07:24:35.691008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:24:35.690978    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d75c53_9177_4bbb_a6f8_4f5d358083c2.slice/crio-e42df279ed471a2b3d47d5dd77fd4885dbb092db14d241cfeb5f267ecc441dc9 WatchSource:0}: Error finding container e42df279ed471a2b3d47d5dd77fd4885dbb092db14d241cfeb5f267ecc441dc9: Status 404 returned error can't find the container with id e42df279ed471a2b3d47d5dd77fd4885dbb092db14d241cfeb5f267ecc441dc9
Nov 02 07:24:35.731061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:35.731032    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cljfx"
Nov 02 07:24:36.511868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:36.511824    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs52m" event={"ID":"e5d75c53-9177-4bbb-a6f8-4f5d358083c2","Type":"ContainerStarted","Data":"e42df279ed471a2b3d47d5dd77fd4885dbb092db14d241cfeb5f267ecc441dc9"}
Nov 02 07:24:37.517583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:37.517538    4515 generic.go:356] "Generic (PLEG): container finished" podID="e5d75c53-9177-4bbb-a6f8-4f5d358083c2" containerID="51f027ab7029a8841e55121baf113c522734edbcebaaa4dca42c40423ed5033e" exitCode=0
Nov 02 07:24:37.518071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:37.517629    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs52m" event={"ID":"e5d75c53-9177-4bbb-a6f8-4f5d358083c2","Type":"ContainerDied","Data":"51f027ab7029a8841e55121baf113c522734edbcebaaa4dca42c40423ed5033e"}
Nov 02 07:24:38.524643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:38.524611    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs52m" event={"ID":"e5d75c53-9177-4bbb-a6f8-4f5d358083c2","Type":"ContainerStarted","Data":"caf4753395416537973624cf92b79173f88784c7146443801e05ed6ff56edf34"}
Nov 02 07:24:38.525068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:38.524670    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rs52m" event={"ID":"e5d75c53-9177-4bbb-a6f8-4f5d358083c2","Type":"ContainerStarted","Data":"6f2170cf0161f2544f236d365900bef33dc6fd9517593f86d3c1a2621a84f13a"}
Nov 02 07:24:38.599135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:38.599076    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rs52m" podStartSLOduration=2.483223753 podStartE2EDuration="3.599060587s" totalImagesPullingTime="1.035769109s" totalInitContainerRuntime="80.067725ms" isStatefulPod=true podCreationTimestamp="2026-11-02 07:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:38.585899879 +0000 UTC m=+23.348336778" watchObservedRunningTime="2026-11-02 07:24:38.599060587 +0000 UTC m=+23.361497509"
Nov 02 07:24:39.313745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:39.313703    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:39.313943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:39.313778    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:39.320523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:39.320493    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:39.532764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:39.532736    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:24:44.548764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:44.548717    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l9ngq" event={"ID":"02f3384d-e450-4445-929f-ea4b686859d9","Type":"ContainerStarted","Data":"73097e2d0e37c733bc0229d777205291c757ac41e36e39e68ca53494086be1c4"}
Nov 02 07:24:44.549264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:44.548938    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:44.564719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:44.564688    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-l9ngq"
Nov 02 07:24:44.659856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:44.659800    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-l9ngq" podStartSLOduration=1.83189717 podStartE2EDuration="18.65977998s" totalImagesPullingTime="16.82788281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:44.617027907 +0000 UTC m=+29.379464814" watchObservedRunningTime="2026-11-02 07:24:44.65977998 +0000 UTC m=+29.422216886"
Nov 02 07:24:47.727484 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:24:47.727441    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:24:48.476837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:48.476798    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56f58c559d-klv5k"
Nov 02 07:24:50.306828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:50.306797    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xq8m6"
Nov 02 07:24:52.934966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:24:52.934930    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:25:17.721592 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:25:17.721552    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:25:17.954687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:17.954620    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-55ff76bb47-f5v8j" podUID="394adac3-7bac-448a-b05d-858af9a0b9f2" containerName="console" containerID="cri-o://3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9" gracePeriod=15
Nov 02 07:25:18.221385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.221344    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:25:18.291945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.291910    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.291945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.291949    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.291976    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292013    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292042    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\" (UniqueName: \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\") pod \"394adac3-7bac-448a-b05d-858af9a0b9f2\" (UID: \"394adac3-7bac-448a-b05d-858af9a0b9f2\") "
Nov 02 07:25:18.292460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292206    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:25:18.292517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292475    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:18.292517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292497    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:18.292595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.292512    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:18.294413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.294383    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:25:18.294529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.294412    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:25:18.294617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.294599    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8" pod "394adac3-7bac-448a-b05d-858af9a0b9f2" (UID: "394adac3-7bac-448a-b05d-858af9a0b9f2"). InnerVolumeSpecName "kube-api-access-hsst8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:25:18.392988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.392955    4515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-serving-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.392988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.392983    4515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-console-config\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.392988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.392993    4515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-oauth-serving-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.393219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.393003    4515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394adac3-7bac-448a-b05d-858af9a0b9f2-console-oauth-config\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.393219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.393013    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsst8\" (UniqueName: \"kubernetes.io/projected/394adac3-7bac-448a-b05d-858af9a0b9f2-kube-api-access-hsst8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.393219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.393022    4515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394adac3-7bac-448a-b05d-858af9a0b9f2-tmp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.393219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.393030    4515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394adac3-7bac-448a-b05d-858af9a0b9f2-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:18.666986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.666885    4515 generic.go:356] "Generic (PLEG): container finished" podID="394adac3-7bac-448a-b05d-858af9a0b9f2" containerID="3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9" exitCode=0
Nov 02 07:25:18.666986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.666930    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff76bb47-f5v8j"
Nov 02 07:25:18.666986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.666969    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff76bb47-f5v8j" event={"ID":"394adac3-7bac-448a-b05d-858af9a0b9f2","Type":"ContainerDied","Data":"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"}
Nov 02 07:25:18.667278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.667032    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff76bb47-f5v8j" event={"ID":"394adac3-7bac-448a-b05d-858af9a0b9f2","Type":"ContainerDied","Data":"6ec8ae4ed5314d96a52179d4926adac112df613fc3f360cc3f3e91409d6f29d3"}
Nov 02 07:25:18.667278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.667059    4515 scope.go:122] "RemoveContainer" containerID="3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"
Nov 02 07:25:18.675688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.675664    4515 scope.go:122] "RemoveContainer" containerID="3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"
Nov 02 07:25:18.675981 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:25:18.675951    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9\": container with ID starting with 3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9 not found: ID does not exist" containerID="3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"
Nov 02 07:25:18.676026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.675991    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9"} err="failed to get container status \"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9\": rpc error: code = NotFound desc = could not find container \"3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9\": container with ID starting with 3407c659450bd4e0051fc1b231331b56a131bfb3474c72a674949910b1438cb9 not found: ID does not exist"
Nov 02 07:25:18.717320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.717263    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:25:18.732072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:18.732040    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55ff76bb47-f5v8j"]
Nov 02 07:25:19.370607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:19.370571    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394adac3-7bac-448a-b05d-858af9a0b9f2" path="/var/lib/kubelet/pods/394adac3-7bac-448a-b05d-858af9a0b9f2/volumes"
Nov 02 07:25:25.525530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.525497    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:25:25.526063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.525706    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" podUID="53590114-7b6a-4223-8361-f82630bb6d5d" containerName="insights-gathering" containerID="cri-o://5ffe369fa0fc40aceb601747c614102166f395af26e2b0c758c9012a732d4cc5" gracePeriod=30
Nov 02 07:25:25.691878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.691844    4515 generic.go:356] "Generic (PLEG): container finished" podID="53590114-7b6a-4223-8361-f82630bb6d5d" containerID="5ffe369fa0fc40aceb601747c614102166f395af26e2b0c758c9012a732d4cc5" exitCode=2
Nov 02 07:25:25.692026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.691912    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" event={"ID":"53590114-7b6a-4223-8361-f82630bb6d5d","Type":"ContainerDied","Data":"5ffe369fa0fc40aceb601747c614102166f395af26e2b0c758c9012a732d4cc5"}
Nov 02 07:25:25.714155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.714123    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gnldr-bgflx"]
Nov 02 07:25:25.714463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.714447    4515 cpu_manager.go:414] "containerMap: removing container" podUID="394adac3-7bac-448a-b05d-858af9a0b9f2" containerName="console"
Nov 02 07:25:25.714547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.714465    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394adac3-7bac-448a-b05d-858af9a0b9f2" containerName="console"
Nov 02 07:25:25.714547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.714543    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="394adac3-7bac-448a-b05d-858af9a0b9f2" containerName="console"
Nov 02 07:25:25.733767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.733738    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.736984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.736954    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gnldr-bgflx"]
Nov 02 07:25:25.807919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.807891    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:25:25.851712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.851667    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqx6v\" (UniqueName: \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.851712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.851714    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.851964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.851736    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.851964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.851849    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.952420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952388    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") pod \"53590114-7b6a-4223-8361-f82630bb6d5d\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") "
Nov 02 07:25:25.952614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952437    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") pod \"53590114-7b6a-4223-8361-f82630bb6d5d\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") "
Nov 02 07:25:25.952614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952462    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") pod \"53590114-7b6a-4223-8361-f82630bb6d5d\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") "
Nov 02 07:25:25.952614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952505    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") pod \"53590114-7b6a-4223-8361-f82630bb6d5d\" (UID: \"53590114-7b6a-4223-8361-f82630bb6d5d\") "
Nov 02 07:25:25.952614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952597    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqx6v\" (UniqueName: \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.952846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952642    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.952846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952666    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.952846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952720    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.952846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952796    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle" pod "53590114-7b6a-4223-8361-f82630bb6d5d" (UID: "53590114-7b6a-4223-8361-f82630bb6d5d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:25.953051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.952892    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle" pod "53590114-7b6a-4223-8361-f82630bb6d5d" (UID: "53590114-7b6a-4223-8361-f82630bb6d5d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:25.953111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.953091    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.953374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.953354    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.953669 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.953651    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path" pod "53590114-7b6a-4223-8361-f82630bb6d5d" (UID: "53590114-7b6a-4223-8361-f82630bb6d5d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:25:25.953944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.953919    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:25.955035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.955017    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf" pod "53590114-7b6a-4223-8361-f82630bb6d5d" (UID: "53590114-7b6a-4223-8361-f82630bb6d5d"). InnerVolumeSpecName "kube-api-access-vmqzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:25:25.976710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:25.976682    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqx6v\" (UniqueName: \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\") pod \"periodic-gathering-gnldr-bgflx\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") " pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:26.053707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.053628    4515 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-service-ca-bundle\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:26.053707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.053656    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqzf\" (UniqueName: \"kubernetes.io/projected/53590114-7b6a-4223-8361-f82630bb6d5d-kube-api-access-vmqzf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:26.053707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.053667    4515 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/53590114-7b6a-4223-8361-f82630bb6d5d-archives-path\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:26.053707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.053675    4515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53590114-7b6a-4223-8361-f82630bb6d5d-trusted-ca-bundle\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:25:26.063629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.063599    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:25:26.391339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.391294    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gnldr-bgflx"]
Nov 02 07:25:26.392685 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:25:26.392658    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80210f00_c605_4553_be91_b6abca06d707.slice/crio-501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc WatchSource:0}: Error finding container 501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc: Status 404 returned error can't find the container with id 501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc
Nov 02 07:25:26.696268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.696192    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6"
Nov 02 07:25:26.696268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.696195    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8qx5v-wgjd6" event={"ID":"53590114-7b6a-4223-8361-f82630bb6d5d","Type":"ContainerDied","Data":"0410ad40cfac516b86f427f0918be1cd85ff058a7f4c0f7e6325fe5b2b421be3"}
Nov 02 07:25:26.696268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.696266    4515 scope.go:122] "RemoveContainer" containerID="5ffe369fa0fc40aceb601747c614102166f395af26e2b0c758c9012a732d4cc5"
Nov 02 07:25:26.698833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.698811    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gnldr-bgflx" event={"ID":"80210f00-c605-4553-be91-b6abca06d707","Type":"ContainerStarted","Data":"db5cff9adf7fb9f69928202ce172997066ac4ef38374df571e3ccacf667ebae4"}
Nov 02 07:25:26.698966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.698851    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gnldr-bgflx" event={"ID":"80210f00-c605-4553-be91-b6abca06d707","Type":"ContainerStarted","Data":"501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc"}
Nov 02 07:25:26.753537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.753488    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gnldr-bgflx" podStartSLOduration=1.753475334 podStartE2EDuration="1.753475334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:25:26.752516427 +0000 UTC m=+71.514953332" watchObservedRunningTime="2026-11-02 07:25:26.753475334 +0000 UTC m=+71.515912240"
Nov 02 07:25:26.787377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.787341    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:25:26.791573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:26.791544    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-8qx5v-wgjd6"]
Nov 02 07:25:27.296362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.296329    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:25:27.296608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.296597    4515 cpu_manager.go:414] "containerMap: removing container" podUID="53590114-7b6a-4223-8361-f82630bb6d5d" containerName="insights-gathering"
Nov 02 07:25:27.296656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.296610    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53590114-7b6a-4223-8361-f82630bb6d5d" containerName="insights-gathering"
Nov 02 07:25:27.296687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.296660    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="53590114-7b6a-4223-8361-f82630bb6d5d" containerName="insights-gathering"
Nov 02 07:25:27.324793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.324762    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:25:27.324962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.324879    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.340463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.340435    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 07:25:27.352670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.352637    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jcgqg\""
Nov 02 07:25:27.352779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.352672    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 07:25:27.352779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.352704    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 07:25:27.352779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.352704    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 07:25:27.352912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.352902    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 07:25:27.370183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.370152    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53590114-7b6a-4223-8361-f82630bb6d5d" path="/var/lib/kubelet/pods/53590114-7b6a-4223-8361-f82630bb6d5d/volumes"
Nov 02 07:25:27.378763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.378742    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 07:25:27.465677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465635    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.465866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465713    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.465866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465736    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.465866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465753    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.465866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465818    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.465866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465855    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.466026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465881    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2m7\" (UniqueName: \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.466026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.465980    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567000    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567072    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567089    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567127    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567144    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567162    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2m7\" (UniqueName: \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567204    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567672 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567633    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567932    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.567955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.567946    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.568084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.568014    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.568229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.568211    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.570122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.570097    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.570122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.570107    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.591793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.591772    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2m7\" (UniqueName: \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\") pod \"console-57475fb996-9nk6t\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") " pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.647679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.647643    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:27.979762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:27.979739    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:25:27.980738 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:25:27.980713    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb45f393_49ac_4994_aa49_40a371e0a2de.slice/crio-fb33c2182d590ed62d98d260f2b6882817c4c08c9e262b2fbe24a12ef72c9248 WatchSource:0}: Error finding container fb33c2182d590ed62d98d260f2b6882817c4c08c9e262b2fbe24a12ef72c9248: Status 404 returned error can't find the container with id fb33c2182d590ed62d98d260f2b6882817c4c08c9e262b2fbe24a12ef72c9248
Nov 02 07:25:28.708800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:28.708760    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57475fb996-9nk6t" event={"ID":"db45f393-49ac-4994-aa49-40a371e0a2de","Type":"ContainerStarted","Data":"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"}
Nov 02 07:25:28.708800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:28.708810    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57475fb996-9nk6t" event={"ID":"db45f393-49ac-4994-aa49-40a371e0a2de","Type":"ContainerStarted","Data":"fb33c2182d590ed62d98d260f2b6882817c4c08c9e262b2fbe24a12ef72c9248"}
Nov 02 07:25:28.752685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:28.752633    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57475fb996-9nk6t" podStartSLOduration=1.752620146 podStartE2EDuration="1.752620146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:25:28.751774057 +0000 UTC m=+73.514210977" watchObservedRunningTime="2026-11-02 07:25:28.752620146 +0000 UTC m=+73.515057046"
Nov 02 07:25:37.648090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:37.648048    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:37.648583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:37.648107    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:37.652780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:37.652757    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:37.744035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:25:37.744009    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:25:47.721438 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:25:47.721399    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:26:02.038945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.038909    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5474bc87-l8b5v"]
Nov 02 07:26:02.040991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.040962    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.063890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.063862    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5474bc87-l8b5v"]
Nov 02 07:26:02.117483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117447    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-oauth-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117506    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-console-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117602    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9b444e-b160-4070-8860-850170ffb81a-tmp\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117640    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-service-ca\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117656    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-oauth-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117676    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-trusted-ca-bundle\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.117887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.117766    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/8d9b444e-b160-4070-8860-850170ffb81a-kube-api-access-tcwkl\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218076    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218122    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-oauth-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218140    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-console-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218191    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9b444e-b160-4070-8860-850170ffb81a-tmp\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218226    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-service-ca\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218362    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-oauth-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218418    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-trusted-ca-bundle\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218454    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/8d9b444e-b160-4070-8860-850170ffb81a-kube-api-access-tcwkl\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.218841 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.218816    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d9b444e-b160-4070-8860-850170ffb81a-tmp\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.219124 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.219100    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-console-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.219191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.219123    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-service-ca\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.219238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.219213    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-oauth-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.219442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.219418    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9b444e-b160-4070-8860-850170ffb81a-trusted-ca-bundle\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.220700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.220680    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-oauth-config\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.220814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.220794    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9b444e-b160-4070-8860-850170ffb81a-console-serving-cert\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.242594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.242562    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/8d9b444e-b160-4070-8860-850170ffb81a-kube-api-access-tcwkl\") pod \"console-5f5474bc87-l8b5v\" (UID: \"8d9b444e-b160-4070-8860-850170ffb81a\") " pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.351022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.350925    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:02.647636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.647603    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5474bc87-l8b5v"]
Nov 02 07:26:02.648668 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:26:02.648641    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9b444e_b160_4070_8860_850170ffb81a.slice/crio-5d7bc3285d0e159e3f287fa00f849af61ca53b7a0be645c60b1724c80f6b95b1 WatchSource:0}: Error finding container 5d7bc3285d0e159e3f287fa00f849af61ca53b7a0be645c60b1724c80f6b95b1: Status 404 returned error can't find the container with id 5d7bc3285d0e159e3f287fa00f849af61ca53b7a0be645c60b1724c80f6b95b1
Nov 02 07:26:02.823705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.823661    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5474bc87-l8b5v" event={"ID":"8d9b444e-b160-4070-8860-850170ffb81a","Type":"ContainerStarted","Data":"ebb940c13adba952f2af8689b09a5efd85466428089bddceea5905615a2b9eca"}
Nov 02 07:26:02.823863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.823717    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5474bc87-l8b5v" event={"ID":"8d9b444e-b160-4070-8860-850170ffb81a","Type":"ContainerStarted","Data":"5d7bc3285d0e159e3f287fa00f849af61ca53b7a0be645c60b1724c80f6b95b1"}
Nov 02 07:26:02.868417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:02.868369    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f5474bc87-l8b5v" podStartSLOduration=0.868354052 podStartE2EDuration="868.354052ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:26:02.867407033 +0000 UTC m=+107.629843940" watchObservedRunningTime="2026-11-02 07:26:02.868354052 +0000 UTC m=+107.630790958"
Nov 02 07:26:12.351618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:12.351573    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:12.351618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:12.351630    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:12.356532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:12.356509    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:12.860996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:12.860968    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5474bc87-l8b5v"
Nov 02 07:26:12.934821 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:12.934787    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:26:17.721882 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:26:17.721837    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:26:37.954758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:37.954667    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57475fb996-9nk6t" podUID="db45f393-49ac-4994-aa49-40a371e0a2de" containerName="console" containerID="cri-o://b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc" gracePeriod=15
Nov 02 07:26:38.199034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.199008    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:26:38.287886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.287849    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.287898    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.287922    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.287953    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.287980    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288008    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\" (UniqueName: \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288073    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288111    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\") pod \"db45f393-49ac-4994-aa49-40a371e0a2de\" (UID: \"db45f393-49ac-4994-aa49-40a371e0a2de\") "
Nov 02 07:26:38.288268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288223    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:26:38.288441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288374    4515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db45f393-49ac-4994-aa49-40a371e0a2de-tmp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.288491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288413    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:38.288622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288584    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:38.288622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288601    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:38.288622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.288593    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:38.290111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.290088    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:26:38.290318 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.290280    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:26:38.290405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.290338    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7" pod "db45f393-49ac-4994-aa49-40a371e0a2de" (UID: "db45f393-49ac-4994-aa49-40a371e0a2de"). InnerVolumeSpecName "kube-api-access-hf2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:26:38.389289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389236    4515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389284    4515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-oauth-config\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389295    4515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db45f393-49ac-4994-aa49-40a371e0a2de-console-serving-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389305    4515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-trusted-ca-bundle\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389335    4515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-console-config\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389344    4515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db45f393-49ac-4994-aa49-40a371e0a2de-oauth-serving-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.389540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.389352    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf2m7\" (UniqueName: \"kubernetes.io/projected/db45f393-49ac-4994-aa49-40a371e0a2de-kube-api-access-hf2m7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:38.945433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.945392    4515 generic.go:356] "Generic (PLEG): container finished" podID="db45f393-49ac-4994-aa49-40a371e0a2de" containerID="b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc" exitCode=0
Nov 02 07:26:38.945641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.945442    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57475fb996-9nk6t"
Nov 02 07:26:38.945641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.945479    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57475fb996-9nk6t" event={"ID":"db45f393-49ac-4994-aa49-40a371e0a2de","Type":"ContainerDied","Data":"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"}
Nov 02 07:26:38.945641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.945533    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57475fb996-9nk6t" event={"ID":"db45f393-49ac-4994-aa49-40a371e0a2de","Type":"ContainerDied","Data":"fb33c2182d590ed62d98d260f2b6882817c4c08c9e262b2fbe24a12ef72c9248"}
Nov 02 07:26:38.945641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.945554    4515 scope.go:122] "RemoveContainer" containerID="b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"
Nov 02 07:26:38.954225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.954210    4515 scope.go:122] "RemoveContainer" containerID="b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"
Nov 02 07:26:38.954575 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:26:38.954556    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc\": container with ID starting with b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc not found: ID does not exist" containerID="b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"
Nov 02 07:26:38.954649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.954584    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc"} err="failed to get container status \"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc\": rpc error: code = NotFound desc = could not find container \"b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc\": container with ID starting with b05a755d5badcce52b7d1cdbfc236cf3b692ffc21df0bf0c8ea83f5d81bce6dc not found: ID does not exist"
Nov 02 07:26:38.997636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:38.997604    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:26:39.010102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:39.010073    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57475fb996-9nk6t"]
Nov 02 07:26:39.369708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:39.369673    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db45f393-49ac-4994-aa49-40a371e0a2de" path="/var/lib/kubelet/pods/db45f393-49ac-4994-aa49-40a371e0a2de/volumes"
Nov 02 07:26:47.721514 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:26:47.721479    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:26:50.988251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:50.988216    4515 generic.go:356] "Generic (PLEG): container finished" podID="80210f00-c605-4553-be91-b6abca06d707" containerID="db5cff9adf7fb9f69928202ce172997066ac4ef38374df571e3ccacf667ebae4" exitCode=0
Nov 02 07:26:50.988651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:50.988270    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gnldr-bgflx" event={"ID":"80210f00-c605-4553-be91-b6abca06d707","Type":"ContainerDied","Data":"db5cff9adf7fb9f69928202ce172997066ac4ef38374df571e3ccacf667ebae4"}
Nov 02 07:26:52.108374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.108300    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:26:52.194612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.194576    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\") pod \"80210f00-c605-4553-be91-b6abca06d707\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") "
Nov 02 07:26:52.194791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.194647    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\" (UniqueName: \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\") pod \"80210f00-c605-4553-be91-b6abca06d707\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") "
Nov 02 07:26:52.194791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.194681    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\") pod \"80210f00-c605-4553-be91-b6abca06d707\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") "
Nov 02 07:26:52.194791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.194730    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\") pod \"80210f00-c605-4553-be91-b6abca06d707\" (UID: \"80210f00-c605-4553-be91-b6abca06d707\") "
Nov 02 07:26:52.195059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.195033    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle" pod "80210f00-c605-4553-be91-b6abca06d707" (UID: "80210f00-c605-4553-be91-b6abca06d707"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:52.195099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.195050    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path" pod "80210f00-c605-4553-be91-b6abca06d707" (UID: "80210f00-c605-4553-be91-b6abca06d707"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:26:52.195162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.195142    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle" pod "80210f00-c605-4553-be91-b6abca06d707" (UID: "80210f00-c605-4553-be91-b6abca06d707"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:26:52.196872 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.196847    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v" pod "80210f00-c605-4553-be91-b6abca06d707" (UID: "80210f00-c605-4553-be91-b6abca06d707"). InnerVolumeSpecName "kube-api-access-kqx6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:26:52.295412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.295378    4515 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-service-ca-bundle\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:52.295412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.295407    4515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80210f00-c605-4553-be91-b6abca06d707-trusted-ca-bundle\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:52.295412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.295417    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqx6v\" (UniqueName: \"kubernetes.io/projected/80210f00-c605-4553-be91-b6abca06d707-kube-api-access-kqx6v\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:52.295638 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:52.295426    4515 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80210f00-c605-4553-be91-b6abca06d707-archives-path\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:26:53.000542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:53.000498    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gnldr-bgflx" event={"ID":"80210f00-c605-4553-be91-b6abca06d707","Type":"ContainerDied","Data":"501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc"}
Nov 02 07:26:53.000542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:53.000539    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gnldr-bgflx"
Nov 02 07:26:53.000757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:26:53.000582    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501b386d703010ffccbba077a1ba1a817d10b0ff7e41b06d72cb8de54883e6cc"
Nov 02 07:27:17.722037 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:27:17.721998    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:27:47.721400 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:27:47.721357    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:28:17.720729 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:28:17.720648    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:28:41.633854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.633825    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634073    4515 cpu_manager.go:414] "containerMap: removing container" podUID="80210f00-c605-4553-be91-b6abca06d707" containerName="insights-gathering"
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634083    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80210f00-c605-4553-be91-b6abca06d707" containerName="insights-gathering"
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634094    4515 cpu_manager.go:414] "containerMap: removing container" podUID="db45f393-49ac-4994-aa49-40a371e0a2de" containerName="console"
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634099    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db45f393-49ac-4994-aa49-40a371e0a2de" containerName="console"
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634143    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="80210f00-c605-4553-be91-b6abca06d707" containerName="insights-gathering"
Nov 02 07:28:41.636171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.634151    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="db45f393-49ac-4994-aa49-40a371e0a2de" containerName="console"
Nov 02 07:28:41.637150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.637134    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.652607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.652582    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 07:28:41.652607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.652584    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 07:28:41.652795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.652658    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 07:28:41.652795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.652664    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 07:28:41.652795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.652714    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 07:28:41.665133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.665110    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 07:28:41.665269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.665159    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bq7lr\""
Nov 02 07:28:41.666380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.666363    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:28:41.788282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788245    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788300    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788343    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788366    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzn8\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788421    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788451    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.788681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.788554    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889363    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889407    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889433    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzn8\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889472    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889499    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889537    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.889716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.889664    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.890130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.890099    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.891850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.891818    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.891979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.891885    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.891979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.891955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.892064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.891952    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.914519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.914490    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.914699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.914680    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzn8\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\") pod \"istiod-openshift-gateway-7876bfcb74-cvhcv\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:41.948156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:41.948121    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:42.263902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:42.263871    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:28:42.264709 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:28:42.264683    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b704ff_6dd3_4179_890d_500a6022fe0f.slice/crio-06ad95f615287ed9e627f72b8064587e2b747ff5391879976cfe54ca0e992018 WatchSource:0}: Error finding container 06ad95f615287ed9e627f72b8064587e2b747ff5391879976cfe54ca0e992018: Status 404 returned error can't find the container with id 06ad95f615287ed9e627f72b8064587e2b747ff5391879976cfe54ca0e992018
Nov 02 07:28:42.341708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:42.341675    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" event={"ID":"90b704ff-6dd3-4179-890d-500a6022fe0f","Type":"ContainerStarted","Data":"06ad95f615287ed9e627f72b8064587e2b747ff5391879976cfe54ca0e992018"}
Nov 02 07:28:44.807203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:44.807164    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:44.807614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:44.807245    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:45.355833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:45.355767    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" event={"ID":"90b704ff-6dd3-4179-890d-500a6022fe0f","Type":"ContainerStarted","Data":"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"}
Nov 02 07:28:45.356803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:45.356778    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:45.361784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:45.361156    4515 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cvhcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:28:45.361784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:45.361218    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:28:45.431594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:45.431531    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" podStartSLOduration=1.891665164 podStartE2EDuration="4.431513542s" totalImagesPullingTime="2.539848378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:28:45.423256859 +0000 UTC m=+270.185693765" watchObservedRunningTime="2026-11-02 07:28:45.431513542 +0000 UTC m=+270.193950449"
Nov 02 07:28:46.359361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:46.359325    4515 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cvhcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:28:46.359738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:46.359380    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:28:47.363173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:28:47.363140    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:28:47.721606 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:28:47.721503    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:29:17.721513 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:29:17.721472    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:29:38.798234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.798153    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"]
Nov 02 07:29:38.801407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.801389    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.818618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.818593    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"]
Nov 02 07:29:38.881475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.881440    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 07:29:38.881661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.881554    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jkbxf\""
Nov 02 07:29:38.881661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.881560    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 07:29:38.881661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.881589    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 07:29:38.882542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.882514    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 07:29:38.882542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.882515    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 07:29:38.885962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.885944    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.886021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.885969    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.886021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.885986    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxlc\" (UniqueName: \"kubernetes.io/projected/fd6cafa4-7c51-4d43-8468-2096bd9355b8-kube-api-access-bdxlc\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.886021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.886015    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.886151 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.886083    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.886230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.886213    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986701    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986738    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986758    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxlc\" (UniqueName: \"kubernetes.io/projected/fd6cafa4-7c51-4d43-8468-2096bd9355b8-kube-api-access-bdxlc\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986795    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986823    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.986984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.986877    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.987192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.987124    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.989095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.989067    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd6cafa4-7c51-4d43-8468-2096bd9355b8-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.989368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.989352    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.989443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.989425    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:38.989548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:38.989528    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd6cafa4-7c51-4d43-8468-2096bd9355b8-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:39.016724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:39.016697    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxlc\" (UniqueName: \"kubernetes.io/projected/fd6cafa4-7c51-4d43-8468-2096bd9355b8-kube-api-access-bdxlc\") pod \"opendatahub-operator-controller-manager-756f6c987b-9lkb5\" (UID: \"fd6cafa4-7c51-4d43-8468-2096bd9355b8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:39.116462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:39.116376    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:39.464460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:39.464434    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"]
Nov 02 07:29:39.465292 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:29:39.465268    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6cafa4_7c51_4d43_8468_2096bd9355b8.slice/crio-0321b8e9eea70543335f9e1f486f08ac3d899fe091c4d596e055134d265b6e6f WatchSource:0}: Error finding container 0321b8e9eea70543335f9e1f486f08ac3d899fe091c4d596e055134d265b6e6f: Status 404 returned error can't find the container with id 0321b8e9eea70543335f9e1f486f08ac3d899fe091c4d596e055134d265b6e6f
Nov 02 07:29:39.467080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:39.467061    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:29:39.540169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:39.540132    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5" event={"ID":"fd6cafa4-7c51-4d43-8468-2096bd9355b8","Type":"ContainerStarted","Data":"0321b8e9eea70543335f9e1f486f08ac3d899fe091c4d596e055134d265b6e6f"}
Nov 02 07:29:42.553969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:42.553931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5" event={"ID":"fd6cafa4-7c51-4d43-8468-2096bd9355b8","Type":"ContainerStarted","Data":"023bdbf93821efc09e59dba9b51e8dfb917ee33c595a116de536f74393d1fc98"}
Nov 02 07:29:43.559073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:43.559037    4515 generic.go:356] "Generic (PLEG): container finished" podID="fd6cafa4-7c51-4d43-8468-2096bd9355b8" containerID="023bdbf93821efc09e59dba9b51e8dfb917ee33c595a116de536f74393d1fc98" exitCode=0
Nov 02 07:29:43.559496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:43.559126    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5" event={"ID":"fd6cafa4-7c51-4d43-8468-2096bd9355b8","Type":"ContainerDied","Data":"023bdbf93821efc09e59dba9b51e8dfb917ee33c595a116de536f74393d1fc98"}
Nov 02 07:29:44.564408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:44.564374    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5" event={"ID":"fd6cafa4-7c51-4d43-8468-2096bd9355b8","Type":"ContainerStarted","Data":"c41b347581e344871fce7acccfb7827ee008811cac9888d5f7d465d23e1d11c0"}
Nov 02 07:29:44.564917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:44.564511    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:29:44.671933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:44.671883    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5" podStartSLOduration=2.95361125 podStartE2EDuration="6.671869805s" totalImagesPullingTime="3.261888413s" totalInitContainerRuntime="456.370142ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:44.659716609 +0000 UTC m=+329.422153516" watchObservedRunningTime="2026-11-02 07:29:44.671869805 +0000 UTC m=+329.434306711"
Nov 02 07:29:47.721157 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:29:47.721107    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:29:55.570502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:29:55.570473    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-9lkb5"
Nov 02 07:30:07.118256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.118220    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"]
Nov 02 07:30:07.121279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.121259    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.155428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.155394    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 07:30:07.155643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.155406    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 07:30:07.155745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.155493    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jmlf4\""
Nov 02 07:30:07.156595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.156554    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"]
Nov 02 07:30:07.206407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.206373    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9kjm\" (UniqueName: \"kubernetes.io/projected/067d8042-97a7-4a81-83d5-06414565ddca-kube-api-access-w9kjm\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.206407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.206417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067d8042-97a7-4a81-83d5-06414565ddca-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.206630 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.206521    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/067d8042-97a7-4a81-83d5-06414565ddca-tmp\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.309203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.309163    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9kjm\" (UniqueName: \"kubernetes.io/projected/067d8042-97a7-4a81-83d5-06414565ddca-kube-api-access-w9kjm\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.309405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.309280    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067d8042-97a7-4a81-83d5-06414565ddca-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.309405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.309360    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/067d8042-97a7-4a81-83d5-06414565ddca-tmp\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.312623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.312590    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/067d8042-97a7-4a81-83d5-06414565ddca-tmp\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.314803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.314777    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067d8042-97a7-4a81-83d5-06414565ddca-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.333751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.333717    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9kjm\" (UniqueName: \"kubernetes.io/projected/067d8042-97a7-4a81-83d5-06414565ddca-kube-api-access-w9kjm\") pod \"kube-auth-proxy-7c6946f9c7-sjbdk\" (UID: \"067d8042-97a7-4a81-83d5-06414565ddca\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.432673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.432594    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"
Nov 02 07:30:07.729953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:07.729926    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk"]
Nov 02 07:30:07.730890 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:30:07.730852    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067d8042_97a7_4a81_83d5_06414565ddca.slice/crio-cef3c1945f597fb40982424d19cd91ff05329834da08a61993885c0077e0564b WatchSource:0}: Error finding container cef3c1945f597fb40982424d19cd91ff05329834da08a61993885c0077e0564b: Status 404 returned error can't find the container with id cef3c1945f597fb40982424d19cd91ff05329834da08a61993885c0077e0564b
Nov 02 07:30:08.646445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:08.646403    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk" event={"ID":"067d8042-97a7-4a81-83d5-06414565ddca","Type":"ContainerStarted","Data":"cef3c1945f597fb40982424d19cd91ff05329834da08a61993885c0077e0564b"}
Nov 02 07:30:09.500275 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.500243    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:30:09.509072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.509042    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.524084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.524005    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:30:09.534984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.534937    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 07:30:09.534984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.534966    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 07:30:09.535192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.535008    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 07:30:09.591033 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.591000    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:30:09.601531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.601496    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tbgzz\""
Nov 02 07:30:09.628370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.628334    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5wt\" (UniqueName: \"kubernetes.io/projected/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-kube-api-access-rn5wt\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.628545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.628423    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-config-file\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.729561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.729524    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-config-file\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.730098 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.729595    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5wt\" (UniqueName: \"kubernetes.io/projected/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-kube-api-access-rn5wt\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.730268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.730243    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-config-file\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.753479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.753399    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5wt\" (UniqueName: \"kubernetes.io/projected/33c78a7e-3e83-4a53-ab31-75f0ef12ade5-kube-api-access-rn5wt\") pod \"limitador-limitador-7c9b87d574-d5hvn\" (UID: \"33c78a7e-3e83-4a53-ab31-75f0ef12ade5\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:09.828292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:09.828257    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:10.130214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:10.130188    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:30:10.130863 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:30:10.130838    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c78a7e_3e83_4a53_ab31_75f0ef12ade5.slice/crio-17baccfe215667d33d3b6ca92995407ba73796f03038635e4d08d74a9f23d4ea WatchSource:0}: Error finding container 17baccfe215667d33d3b6ca92995407ba73796f03038635e4d08d74a9f23d4ea: Status 404 returned error can't find the container with id 17baccfe215667d33d3b6ca92995407ba73796f03038635e4d08d74a9f23d4ea
Nov 02 07:30:10.656277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:10.656230    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk" event={"ID":"067d8042-97a7-4a81-83d5-06414565ddca","Type":"ContainerStarted","Data":"798b8db77d4b1cfcd0a833571dff706d16f75595159dd64ed13c8b0903c92c4e"}
Nov 02 07:30:10.657541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:10.657514    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn" event={"ID":"33c78a7e-3e83-4a53-ab31-75f0ef12ade5","Type":"ContainerStarted","Data":"17baccfe215667d33d3b6ca92995407ba73796f03038635e4d08d74a9f23d4ea"}
Nov 02 07:30:10.697826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:10.697767    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-sjbdk" podStartSLOduration=1.303695346 podStartE2EDuration="3.697752194s" totalImagesPullingTime="2.394056848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:10.696516495 +0000 UTC m=+355.458953400" watchObservedRunningTime="2026-11-02 07:30:10.697752194 +0000 UTC m=+355.460189099"
Nov 02 07:30:12.666186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:12.666102    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn" event={"ID":"33c78a7e-3e83-4a53-ab31-75f0ef12ade5","Type":"ContainerStarted","Data":"72bae022f5a2d80ccfa1b17d500b9b948b008159de4920a195fb88b9c0edcf96"}
Nov 02 07:30:12.666574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:12.666333    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:12.708711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:12.708663    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn" podStartSLOduration=1.429509157 podStartE2EDuration="3.708648946s" totalImagesPullingTime="2.279139789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:12.70743197 +0000 UTC m=+357.469868876" watchObservedRunningTime="2026-11-02 07:30:12.708648946 +0000 UTC m=+357.471085889"
Nov 02 07:30:17.721357 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:17.721321    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:30:23.670164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:23.670132    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"
Nov 02 07:30:47.721869 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:47.721830    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:30:48.299074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.299039    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:30:48.305349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.305327    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.322780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.322753    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 07:30:48.325383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.325360    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 07:30:48.325511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.325387    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 07:30:48.336680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.336656    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:30:48.338016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.337993    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xfsgv\""
Nov 02 07:30:48.338016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.338010    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 07:30:48.341461 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.341438    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.341557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.341468    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85zj\" (UniqueName: \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.341557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.341503    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.423422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.423394    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 07:30:48.442680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.442648    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.442812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.442687    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85zj\" (UniqueName: \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.442812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.442728    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.442924 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:48.442894    4515 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Nov 02 07:30:48.442978 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:48.442955    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs podName:8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed nodeName:}" failed. No retries permitted until 2026-11-02 07:30:48.942939454 +0000 UTC m=+393.705376342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs") pod "maas-controller-84d6885777-t98mz" (UID: "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed") : secret "maas-controller-metrics-tls" not found
Nov 02 07:30:48.445040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.445017    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.468414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.468384    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85zj\" (UniqueName: \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.814053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.814022    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 07:30:48.816285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.816267    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.839201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.839173    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 07:30:48.845520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.845496    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kk5r\" (UniqueName: \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.845648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.845549    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.845648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.845574    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.946025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.945988    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.946206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.946029    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.946206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.946087    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.946206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.946136    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kk5r\" (UniqueName: \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.948956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.948923    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.949100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.949077    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:48.949174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.949151    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") pod \"maas-controller-84d6885777-t98mz\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") " pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:48.969868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:48.969836    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kk5r\" (UniqueName: \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\") pod \"maas-controller-5f6b5c786d-jb7pn\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") " pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:49.128025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.127937    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:49.217621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.217586    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:49.422389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.422356    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 07:30:49.422798 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:30:49.422774    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21f8235_4923_4a54_98dd_164cac55fadd.slice/crio-80ba6ccb988c8076999bc5e1bbf292bf843004bb56210d521ddee823121c41e7 WatchSource:0}: Error finding container 80ba6ccb988c8076999bc5e1bbf292bf843004bb56210d521ddee823121c41e7: Status 404 returned error can't find the container with id 80ba6ccb988c8076999bc5e1bbf292bf843004bb56210d521ddee823121c41e7
Nov 02 07:30:49.512385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.512357    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:30:49.512980 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:30:49.512957    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8099dbb4_d41a_4fa3_a581_56a3b0b6a3ed.slice/crio-46a6fde52a700313b3c4493e74789ae353e898320b60333167f2188b2f9e79fd WatchSource:0}: Error finding container 46a6fde52a700313b3c4493e74789ae353e898320b60333167f2188b2f9e79fd: Status 404 returned error can't find the container with id 46a6fde52a700313b3c4493e74789ae353e898320b60333167f2188b2f9e79fd
Nov 02 07:30:49.800977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.800940    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d6885777-t98mz" event={"ID":"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed","Type":"ContainerStarted","Data":"46a6fde52a700313b3c4493e74789ae353e898320b60333167f2188b2f9e79fd"}
Nov 02 07:30:49.802003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:49.801978    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" event={"ID":"e21f8235-4923-4a54-98dd-164cac55fadd","Type":"ContainerStarted","Data":"80ba6ccb988c8076999bc5e1bbf292bf843004bb56210d521ddee823121c41e7"}
Nov 02 07:30:52.821822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.821782    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d6885777-t98mz" event={"ID":"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed","Type":"ContainerStarted","Data":"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"}
Nov 02 07:30:52.822337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.821875    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:30:52.823195 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.823168    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" event={"ID":"e21f8235-4923-4a54-98dd-164cac55fadd","Type":"ContainerStarted","Data":"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"}
Nov 02 07:30:52.823327 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.823292    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:30:52.871419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.871366    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-84d6885777-t98mz" podStartSLOduration=2.099565843 podStartE2EDuration="4.871352108s" totalImagesPullingTime="2.771786265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:52.8704681 +0000 UTC m=+397.632905006" watchObservedRunningTime="2026-11-02 07:30:52.871352108 +0000 UTC m=+397.633789013"
Nov 02 07:30:52.914569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:52.914519    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" podStartSLOduration=2.176838534 podStartE2EDuration="4.914506099s" totalImagesPullingTime="2.737667565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:52.913692509 +0000 UTC m=+397.676129406" watchObservedRunningTime="2026-11-02 07:30:52.914506099 +0000 UTC m=+397.676943004"
Nov 02 07:30:54.879946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.879914    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"]
Nov 02 07:30:54.883283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.883264    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.897671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.897642    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.897800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.897686    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxq5\" (UniqueName: \"kubernetes.io/projected/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-kube-api-access-vrxq5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.897867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.897803    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.897867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.897860    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.898870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.898854    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 07:30:54.904644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.904621    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 07:30:54.911704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.911667    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mtc47\""
Nov 02 07:30:54.919343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.919302    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"]
Nov 02 07:30:54.998826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.998792    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.998826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.998814    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 07:30:54.998826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.998830    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxq5\" (UniqueName: \"kubernetes.io/projected/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-kube-api-access-vrxq5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.999087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.998865    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.999087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.998890    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:54.999087 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:54.998977    4515 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 02 07:30:54.999087 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:30:54.999042    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source podName:cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df nodeName:}" failed. No retries permitted until 2026-11-02 07:30:55.499026564 +0000 UTC m=+400.261463462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-fj5gs" (UID: "cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df") : configmap "usage-tenancy-proxy-source" not found
Nov 02 07:30:54.999553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:54.999533    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:55.001343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:55.001304    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:55.023051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:55.023023    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxq5\" (UniqueName: \"kubernetes.io/projected/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-kube-api-access-vrxq5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:55.502634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:55.502598    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:55.503385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:55.503365    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5gs\" (UID: \"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:55.794987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:55.794897    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:30:56.092661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:56.092631    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"]
Nov 02 07:30:56.093060 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:30:56.092990    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf2fa24_d0ad_4baa_a3ef_b883fa6ca9df.slice/crio-69747b12a65ecd5ae1f66a1970f32960500dcaf7f0308f5b81e4de304ec496f3 WatchSource:0}: Error finding container 69747b12a65ecd5ae1f66a1970f32960500dcaf7f0308f5b81e4de304ec496f3: Status 404 returned error can't find the container with id 69747b12a65ecd5ae1f66a1970f32960500dcaf7f0308f5b81e4de304ec496f3
Nov 02 07:30:56.840487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:30:56.840443    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs" event={"ID":"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df","Type":"ContainerStarted","Data":"69747b12a65ecd5ae1f66a1970f32960500dcaf7f0308f5b81e4de304ec496f3"}
Nov 02 07:31:03.829379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:03.829280    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 07:31:03.829379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:03.829375    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:31:03.948559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:03.948529    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:31:03.948800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:03.948770    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-84d6885777-t98mz" podUID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" containerName="manager" containerID="cri-o://b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af" gracePeriod=30
Nov 02 07:31:04.201555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.201526    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:31:04.290193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.290151    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") pod \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") "
Nov 02 07:31:04.290425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.290232    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\" (UniqueName: \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\") pod \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") "
Nov 02 07:31:04.290425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.290289    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\") pod \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\" (UID: \"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed\") "
Nov 02 07:31:04.293697 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.293663    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert" pod "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" (UID: "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:04.300182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.300124    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs" pod "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" (UID: "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:04.300182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.300154    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj" pod "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" (UID: "8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed"). InnerVolumeSpecName "kube-api-access-c85zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:04.396738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.396657    4515 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-webhook-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:04.396738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.396691    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:04.396738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.396705    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85zj\" (UniqueName: \"kubernetes.io/projected/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed-kube-api-access-c85zj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:04.877634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.877594    4515 generic.go:356] "Generic (PLEG): container finished" podID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" containerID="b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af" exitCode=0
Nov 02 07:31:04.878115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.877654    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d6885777-t98mz"
Nov 02 07:31:04.878115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.877675    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d6885777-t98mz" event={"ID":"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed","Type":"ContainerDied","Data":"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"}
Nov 02 07:31:04.878115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.877745    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d6885777-t98mz" event={"ID":"8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed","Type":"ContainerDied","Data":"46a6fde52a700313b3c4493e74789ae353e898320b60333167f2188b2f9e79fd"}
Nov 02 07:31:04.878115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.877776    4515 scope.go:122] "RemoveContainer" containerID="b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"
Nov 02 07:31:04.887212 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.887188    4515 scope.go:122] "RemoveContainer" containerID="b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"
Nov 02 07:31:04.887521 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:31:04.887497    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af\": container with ID starting with b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af not found: ID does not exist" containerID="b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"
Nov 02 07:31:04.887591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.887528    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af"} err="failed to get container status \"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af\": rpc error: code = NotFound desc = could not find container \"b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af\": container with ID starting with b5981078b85e61a3ae39e9102cdb0bfcf3e508e0e0db281565aac76a6b7863af not found: ID does not exist"
Nov 02 07:31:04.933597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.933564    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:31:04.938435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:04.938407    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d6885777-t98mz"]
Nov 02 07:31:05.371058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:05.371022    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" path="/var/lib/kubelet/pods/8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed/volumes"
Nov 02 07:31:08.899671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:08.899634    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs" event={"ID":"cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df","Type":"ContainerStarted","Data":"4f989242264338ec736e673f5bca64149c5e599133b4cee6a5d9562490e1b721"}
Nov 02 07:31:08.900187 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:08.899903    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:31:08.941576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:08.941520    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs" podStartSLOduration=3.202330449 podStartE2EDuration="14.94150634s" totalImagesPullingTime="11.739175891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:08.940866618 +0000 UTC m=+413.703303525" watchObservedRunningTime="2026-11-02 07:31:08.94150634 +0000 UTC m=+413.703943237"
Nov 02 07:31:09.068672 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.068628    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 07:31:09.068952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.068940    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" containerName="manager"
Nov 02 07:31:09.068990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.068953    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" containerName="manager"
Nov 02 07:31:09.069023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.069016    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8099dbb4-d41a-4fa3-a581-56a3b0b6a3ed" containerName="manager"
Nov 02 07:31:09.072170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.072147    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.096804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.096759    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 07:31:09.137347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137300    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137359    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137413    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137439    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137470    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137521    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.137711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.137662    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ldq\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238586    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238630    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238655    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238689    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ldq\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238856    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238907    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.238953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.238939    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.239602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.239571    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.240902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.240869    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.241081 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.241063    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.241081 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.241070    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.241198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.241117    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.286059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.286019    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ldq\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.286059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.286047    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z89sk\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.383453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.383415    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.697490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.697454    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 07:31:09.698927 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:31:09.698900    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c34e82d_1916_4a64_b82c_39f022d509a4.slice/crio-7b0418496dbd02c1759f8c21a74748a2d62f08e4d76458d50364150edc4e31e6 WatchSource:0}: Error finding container 7b0418496dbd02c1759f8c21a74748a2d62f08e4d76458d50364150edc4e31e6: Status 404 returned error can't find the container with id 7b0418496dbd02c1759f8c21a74748a2d62f08e4d76458d50364150edc4e31e6
Nov 02 07:31:09.701189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.701159    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:31:09.701255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.701226    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:31:09.906139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.906100    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" event={"ID":"0c34e82d-1916-4a64-b82c-39f022d509a4","Type":"ContainerStarted","Data":"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"}
Nov 02 07:31:09.906689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.906161    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" event={"ID":"0c34e82d-1916-4a64-b82c-39f022d509a4","Type":"ContainerStarted","Data":"7b0418496dbd02c1759f8c21a74748a2d62f08e4d76458d50364150edc4e31e6"}
Nov 02 07:31:09.906689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.906336    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:09.954192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:09.954098    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" podStartSLOduration=0.954083639 podStartE2EDuration="954.083639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:09.953616951 +0000 UTC m=+414.716053857" watchObservedRunningTime="2026-11-02 07:31:09.954083639 +0000 UTC m=+414.716520544"
Nov 02 07:31:10.911895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:10.911864    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 07:31:11.007967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.007933    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:31:11.008186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.008141    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery" containerID="cri-o://eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e" gracePeriod=30
Nov 02 07:31:11.264517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.264491    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:31:11.357729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357681    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357741    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357787    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357808    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357850    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357874    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.357954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.357900    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\") pod \"90b704ff-6dd3-4179-890d-500a6022fe0f\" (UID: \"90b704ff-6dd3-4179-890d-500a6022fe0f\") "
Nov 02 07:31:11.358331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.358277    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:31:11.360204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360167    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:11.360327 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360226    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:11.360447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360422    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:11.360528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360482    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:31:11.360562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360521    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "kube-api-access-bdzn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:11.360627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.360611    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert" pod "90b704ff-6dd3-4179-890d-500a6022fe0f" (UID: "90b704ff-6dd3-4179-890d-500a6022fe0f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459404    4515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b704ff-6dd3-4179-890d-500a6022fe0f-local-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459450    4515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-kubeconfig\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459463    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459473    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-ca-configmap\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459483    4515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-cacerts\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459492    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzn8\" (UniqueName: \"kubernetes.io/projected/90b704ff-6dd3-4179-890d-500a6022fe0f-kube-api-access-bdzn8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.459500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.459501    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b704ff-6dd3-4179-890d-500a6022fe0f-istio-csr-dns-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:11.915184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.915151    4515 generic.go:356] "Generic (PLEG): container finished" podID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerID="eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e" exitCode=0
Nov 02 07:31:11.915644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.915197    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"
Nov 02 07:31:11.915644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.915241    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" event={"ID":"90b704ff-6dd3-4179-890d-500a6022fe0f","Type":"ContainerDied","Data":"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"}
Nov 02 07:31:11.915644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.915300    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv" event={"ID":"90b704ff-6dd3-4179-890d-500a6022fe0f","Type":"ContainerDied","Data":"06ad95f615287ed9e627f72b8064587e2b747ff5391879976cfe54ca0e992018"}
Nov 02 07:31:11.915644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.915341    4515 scope.go:122] "RemoveContainer" containerID="eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"
Nov 02 07:31:11.923366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.923350    4515 scope.go:122] "RemoveContainer" containerID="eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"
Nov 02 07:31:11.923607 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:31:11.923588    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e\": container with ID starting with eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e not found: ID does not exist" containerID="eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"
Nov 02 07:31:11.923649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.923616    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e"} err="failed to get container status \"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e\": rpc error: code = NotFound desc = could not find container \"eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e\": container with ID starting with eb36cac7217f78e3a794a518a0e5d6f3c47cbe216f61c10a38c0990074c6872e not found: ID does not exist"
Nov 02 07:31:11.983653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.983618    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:31:11.996448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:11.996402    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cvhcv"]
Nov 02 07:31:13.371603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:13.371570    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" path="/var/lib/kubelet/pods/90b704ff-6dd3-4179-890d-500a6022fe0f/volumes"
Nov 02 07:31:17.720930 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:31:17.720876    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:31:19.912614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:19.912584    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5gs"
Nov 02 07:31:25.959582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.959548    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:25.960009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.959838    4515 cpu_manager.go:414] "containerMap: removing container" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery"
Nov 02 07:31:25.960009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.959848    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery"
Nov 02 07:31:25.960009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.959915    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b704ff-6dd3-4179-890d-500a6022fe0f" containerName="discovery"
Nov 02 07:31:25.965852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.965828    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:25.982101 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.982076    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 07:31:25.994753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.994723    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:31:25.994924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.994806    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:31:25.994924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.994811    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 07:31:25.996819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:25.996791    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:26.090252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.090208    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.090463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.090255    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.090463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.090284    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.090463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.090433    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvt8\" (UniqueName: \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.191782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.191741    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvt8\" (UniqueName: \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.191977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.191845    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.191977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.191872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.191977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.191894    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.192535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.192499    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.194384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.194357    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.194554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.194531    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.215036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.214966    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvt8\" (UniqueName: \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\") pod \"maas-api-5c66d8d877-97rbw\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") " pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.278870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.278836    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:26.579674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.579644    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:26.581124 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:31:26.581093    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57c0d2d_f3e4_471a_be4e_301473a789d2.slice/crio-640f8f2400cec0e50c87707554cc83cee9363baf7e399cd2caf0a60716c6a378 WatchSource:0}: Error finding container 640f8f2400cec0e50c87707554cc83cee9363baf7e399cd2caf0a60716c6a378: Status 404 returned error can't find the container with id 640f8f2400cec0e50c87707554cc83cee9363baf7e399cd2caf0a60716c6a378
Nov 02 07:31:26.974483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:26.974395    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" event={"ID":"c57c0d2d-f3e4-471a-be4e-301473a789d2","Type":"ContainerStarted","Data":"640f8f2400cec0e50c87707554cc83cee9363baf7e399cd2caf0a60716c6a378"}
Nov 02 07:31:27.980810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:27.980773    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:31:28.984808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:28.984765    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" event={"ID":"c57c0d2d-f3e4-471a-be4e-301473a789d2","Type":"ContainerStarted","Data":"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"}
Nov 02 07:31:28.985215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:28.984896    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:29.027168 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:29.027110    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" podStartSLOduration=2.518150811 podStartE2EDuration="4.027094922s" totalImagesPullingTime="1.508944111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:29.026693972 +0000 UTC m=+433.789130883" watchObservedRunningTime="2026-11-02 07:31:29.027094922 +0000 UTC m=+433.789531824"
Nov 02 07:31:34.994391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:34.994363    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:36.060018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.059986    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 07:31:36.072923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.072891    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.084600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.084575    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 07:31:36.095871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.095840    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4tgq8\""
Nov 02 07:31:36.172104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.172066    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.172260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.172120    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.172302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.172257    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wtj\" (UniqueName: \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.172362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.172330    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.272979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.272938    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.273181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.272993    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.273181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.273037    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wtj\" (UniqueName: \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.273181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.273060    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.273753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.273733    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.275465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.275437    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.275567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.275550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.298005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.297970    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wtj\" (UniqueName: \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\") pod \"maas-api-699d9fc559-k7mqt\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.388582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.388501    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:36.731810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:36.731777    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 07:31:36.732703 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:31:36.732672    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda054e471_aacb_438a_b1c8_f56fdedd3bd2.slice/crio-472d740d0523ec348f6730b5dc1d765232f0fa6c37b9562e056c9aaf06a186cc WatchSource:0}: Error finding container 472d740d0523ec348f6730b5dc1d765232f0fa6c37b9562e056c9aaf06a186cc: Status 404 returned error can't find the container with id 472d740d0523ec348f6730b5dc1d765232f0fa6c37b9562e056c9aaf06a186cc
Nov 02 07:31:37.017445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:37.017411    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" event={"ID":"a054e471-aacb-438a-b1c8-f56fdedd3bd2","Type":"ContainerStarted","Data":"472d740d0523ec348f6730b5dc1d765232f0fa6c37b9562e056c9aaf06a186cc"}
Nov 02 07:31:38.022419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:38.022379    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" event={"ID":"a054e471-aacb-438a-b1c8-f56fdedd3bd2","Type":"ContainerStarted","Data":"c85882f9cf6233caae03f4793a03dcab5a246b37784e70335cfd93744d71789f"}
Nov 02 07:31:38.022826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:38.022533    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:38.065432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:38.065380    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" podStartSLOduration=1.491946381 podStartE2EDuration="2.065346406s" totalImagesPullingTime="573.400025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:38.064886678 +0000 UTC m=+442.827323584" watchObservedRunningTime="2026-11-02 07:31:38.065346406 +0000 UTC m=+442.827783312"
Nov 02 07:31:44.032126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.032097    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 07:31:44.113375 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.113340    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:44.113659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.113622    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" podUID="c57c0d2d-f3e4-471a-be4e-301473a789d2" containerName="maas-api" containerID="cri-o://3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b" gracePeriod=30
Nov 02 07:31:44.358706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.358672    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:44.528018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.527983    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\") pod \"c57c0d2d-f3e4-471a-be4e-301473a789d2\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") "
Nov 02 07:31:44.528209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.528035    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\") pod \"c57c0d2d-f3e4-471a-be4e-301473a789d2\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") "
Nov 02 07:31:44.528209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.528091    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\") pod \"c57c0d2d-f3e4-471a-be4e-301473a789d2\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") "
Nov 02 07:31:44.528209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.528121    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\" (UniqueName: \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\") pod \"c57c0d2d-f3e4-471a-be4e-301473a789d2\" (UID: \"c57c0d2d-f3e4-471a-be4e-301473a789d2\") "
Nov 02 07:31:44.528496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.528465    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca" pod "c57c0d2d-f3e4-471a-be4e-301473a789d2" (UID: "c57c0d2d-f3e4-471a-be4e-301473a789d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:31:44.530262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.530232    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs" pod "c57c0d2d-f3e4-471a-be4e-301473a789d2" (UID: "c57c0d2d-f3e4-471a-be4e-301473a789d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:44.530390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.530253    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8" pod "c57c0d2d-f3e4-471a-be4e-301473a789d2" (UID: "c57c0d2d-f3e4-471a-be4e-301473a789d2"). InnerVolumeSpecName "kube-api-access-xmvt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:44.530390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.530287    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls" pod "c57c0d2d-f3e4-471a-be4e-301473a789d2" (UID: "c57c0d2d-f3e4-471a-be4e-301473a789d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:44.629144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.629059    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57c0d2d-f3e4-471a-be4e-301473a789d2-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:44.629144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.629090    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvt8\" (UniqueName: \"kubernetes.io/projected/c57c0d2d-f3e4-471a-be4e-301473a789d2-kube-api-access-xmvt8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:44.629144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.629101    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:44.629144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:44.629110    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57c0d2d-f3e4-471a-be4e-301473a789d2-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:31:45.051014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.050981    4515 generic.go:356] "Generic (PLEG): container finished" podID="c57c0d2d-f3e4-471a-be4e-301473a789d2" containerID="3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b" exitCode=0
Nov 02 07:31:45.051450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.051030    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"
Nov 02 07:31:45.051450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.051046    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" event={"ID":"c57c0d2d-f3e4-471a-be4e-301473a789d2","Type":"ContainerDied","Data":"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"}
Nov 02 07:31:45.051450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.051086    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw" event={"ID":"c57c0d2d-f3e4-471a-be4e-301473a789d2","Type":"ContainerDied","Data":"640f8f2400cec0e50c87707554cc83cee9363baf7e399cd2caf0a60716c6a378"}
Nov 02 07:31:45.051450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.051107    4515 scope.go:122] "RemoveContainer" containerID="3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"
Nov 02 07:31:45.059538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.059520    4515 scope.go:122] "RemoveContainer" containerID="3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"
Nov 02 07:31:45.059784 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:31:45.059765    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b\": container with ID starting with 3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b not found: ID does not exist" containerID="3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"
Nov 02 07:31:45.059828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.059792    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b"} err="failed to get container status \"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b\": rpc error: code = NotFound desc = could not find container \"3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b\": container with ID starting with 3e882dbd3a6a4a8a7d7af3c04a85e560e0da9689b2b108c17d018dc9f7989d9b not found: ID does not exist"
Nov 02 07:31:45.102779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.102749    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:45.115226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.115198    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66d8d877-97rbw"]
Nov 02 07:31:45.370808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:31:45.370734    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57c0d2d-f3e4-471a-be4e-301473a789d2" path="/var/lib/kubelet/pods/c57c0d2d-f3e4-471a-be4e-301473a789d2/volumes"
Nov 02 07:31:47.721816 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:31:47.721767    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:32:17.721786 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:32:17.721747    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:32:47.722540 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:32:47.722495    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:32:53.027478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.027444    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:32:53.027850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.027757    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c57c0d2d-f3e4-471a-be4e-301473a789d2" containerName="maas-api"
Nov 02 07:32:53.027850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.027768    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57c0d2d-f3e4-471a-be4e-301473a789d2" containerName="maas-api"
Nov 02 07:32:53.027850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.027825    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57c0d2d-f3e4-471a-be4e-301473a789d2" containerName="maas-api"
Nov 02 07:32:53.030780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.030759    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.047023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.046968    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 07:32:53.047156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.047126    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 07:32:53.047202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.047168    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mmbr9\""
Nov 02 07:32:53.061113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.061082    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:32:53.075618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.075572    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.075884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.075856    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpv5\" (UniqueName: \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.177233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.177193    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.177233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.177236    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpv5\" (UniqueName: \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.179732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.179707    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.201041 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.201007    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpv5\" (UniqueName: \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\") pod \"ai-gateway-controller-7d85675f98-p2fzb\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.343673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.343586    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:53.639954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:53.639924    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:32:53.640532 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:32:53.640506    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4410f891_32d9_4658_a366_750e14bc4461.slice/crio-960df09901fe1f83ec2b1104e777d612c14fb4d99a7e57e7f684f1446be9b17a WatchSource:0}: Error finding container 960df09901fe1f83ec2b1104e777d612c14fb4d99a7e57e7f684f1446be9b17a: Status 404 returned error can't find the container with id 960df09901fe1f83ec2b1104e777d612c14fb4d99a7e57e7f684f1446be9b17a
Nov 02 07:32:54.320415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:54.320373    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" event={"ID":"4410f891-32d9-4658-a366-750e14bc4461","Type":"ContainerStarted","Data":"960df09901fe1f83ec2b1104e777d612c14fb4d99a7e57e7f684f1446be9b17a"}
Nov 02 07:32:56.332270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:56.332231    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" event={"ID":"4410f891-32d9-4658-a366-750e14bc4461","Type":"ContainerStarted","Data":"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"}
Nov 02 07:32:56.332762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:56.332302    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:32:56.374954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:56.374899    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" podStartSLOduration=1.047030459 podStartE2EDuration="3.374885553s" totalImagesPullingTime="2.327855094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:32:56.374482899 +0000 UTC m=+521.136919804" watchObservedRunningTime="2026-11-02 07:32:56.374885553 +0000 UTC m=+521.137322459"
Nov 02 07:32:58.212030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.211994    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 07:32:58.215351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.215333    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.233762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.233732    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 07:32:58.243716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.243692    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nhrrv\""
Nov 02 07:32:58.244954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.244933    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 07:32:58.272237 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.272205    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 07:32:58.275491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.275474    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.294828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.294796    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 07:32:58.324996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.324948    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.324996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.325006    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.325283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.325106    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcq6\" (UniqueName: \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.325283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.325158    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52sqg\" (UniqueName: \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.425945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.425908    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.426127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.425969    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcq6\" (UniqueName: \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.426127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.425994    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52sqg\" (UniqueName: \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.426261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.426133    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.426757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.426737    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.426867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.426848    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.449939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.449908    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcq6\" (UniqueName: \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\") pod \"payload-pre-processing-7b98bb678c-6dzmc\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.450067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.450007    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52sqg\" (UniqueName: \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\") pod \"payload-processing-55fbbbf9dc-pwdh9\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.527194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.527154    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:32:58.587670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.587636    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:32:58.822701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.822672    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 07:32:58.823278 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:32:58.823252    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad9dc47_790e_4c20_952e_c2984691c5d4.slice/crio-5128135d0f857be6c752e566db8a1fdc97747778475b98651c6ba4a214e29506 WatchSource:0}: Error finding container 5128135d0f857be6c752e566db8a1fdc97747778475b98651c6ba4a214e29506: Status 404 returned error can't find the container with id 5128135d0f857be6c752e566db8a1fdc97747778475b98651c6ba4a214e29506
Nov 02 07:32:58.886732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:58.886708    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 07:32:58.887425 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:32:58.887399    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5622d69a_45ba_47b1_a91c_ca030d136441.slice/crio-a72829309996c53dcde299812b099b11467716510d74d36f3747911f8d5a3bcb WatchSource:0}: Error finding container a72829309996c53dcde299812b099b11467716510d74d36f3747911f8d5a3bcb: Status 404 returned error can't find the container with id a72829309996c53dcde299812b099b11467716510d74d36f3747911f8d5a3bcb
Nov 02 07:32:59.348026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:59.347987    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" event={"ID":"dad9dc47-790e-4c20-952e-c2984691c5d4","Type":"ContainerStarted","Data":"5128135d0f857be6c752e566db8a1fdc97747778475b98651c6ba4a214e29506"}
Nov 02 07:32:59.349111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:32:59.349074    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" event={"ID":"5622d69a-45ba-47b1-a91c-ca030d136441","Type":"ContainerStarted","Data":"a72829309996c53dcde299812b099b11467716510d74d36f3747911f8d5a3bcb"}
Nov 02 07:33:01.360190 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.360149    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" event={"ID":"dad9dc47-790e-4c20-952e-c2984691c5d4","Type":"ContainerStarted","Data":"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"}
Nov 02 07:33:01.360707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.360234    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:33:01.361640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.361618    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" event={"ID":"5622d69a-45ba-47b1-a91c-ca030d136441","Type":"ContainerStarted","Data":"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"}
Nov 02 07:33:01.361742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.361735    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:33:01.402668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.402565    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" podStartSLOduration=1.082929897 podStartE2EDuration="3.402551574s" totalImagesPullingTime="2.319621677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:01.401511418 +0000 UTC m=+526.163948324" watchObservedRunningTime="2026-11-02 07:33:01.402551574 +0000 UTC m=+526.164988479"
Nov 02 07:33:01.443097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:01.443042    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" podStartSLOduration=1.151824641 podStartE2EDuration="3.443028805s" totalImagesPullingTime="2.291204164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:01.442517526 +0000 UTC m=+526.204954433" watchObservedRunningTime="2026-11-02 07:33:01.443028805 +0000 UTC m=+526.205465710"
Nov 02 07:33:07.338560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.338530    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:33:07.606356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.606273    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-785d49bf89-46p4s"]
Nov 02 07:33:07.609684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.609662    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.629642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.629613    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-785d49bf89-46p4s"]
Nov 02 07:33:07.707814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.707782    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktjm\" (UniqueName: \"kubernetes.io/projected/dc11f2c7-aac8-4213-aa09-42b98173d446-kube-api-access-hktjm\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.708008 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.707830    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11f2c7-aac8-4213-aa09-42b98173d446-metrics-certs\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.808810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.808756    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktjm\" (UniqueName: \"kubernetes.io/projected/dc11f2c7-aac8-4213-aa09-42b98173d446-kube-api-access-hktjm\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.808955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.808827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11f2c7-aac8-4213-aa09-42b98173d446-metrics-certs\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.811238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.811213    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11f2c7-aac8-4213-aa09-42b98173d446-metrics-certs\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.833047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.833015    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktjm\" (UniqueName: \"kubernetes.io/projected/dc11f2c7-aac8-4213-aa09-42b98173d446-kube-api-access-hktjm\") pod \"ai-gateway-controller-785d49bf89-46p4s\" (UID: \"dc11f2c7-aac8-4213-aa09-42b98173d446\") " pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:07.921929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:07.921819    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:08.218401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:08.218370    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-785d49bf89-46p4s"]
Nov 02 07:33:08.219252 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:33:08.219226    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc11f2c7_aac8_4213_aa09_42b98173d446.slice/crio-fbc0aad628ebc495ab111722ce3ba27e74d98a70816f42a132d28f71c28c03d9 WatchSource:0}: Error finding container fbc0aad628ebc495ab111722ce3ba27e74d98a70816f42a132d28f71c28c03d9: Status 404 returned error can't find the container with id fbc0aad628ebc495ab111722ce3ba27e74d98a70816f42a132d28f71c28c03d9
Nov 02 07:33:08.389805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:08.389768    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s" event={"ID":"dc11f2c7-aac8-4213-aa09-42b98173d446","Type":"ContainerStarted","Data":"fbc0aad628ebc495ab111722ce3ba27e74d98a70816f42a132d28f71c28c03d9"}
Nov 02 07:33:09.394581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:09.394543    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s" event={"ID":"dc11f2c7-aac8-4213-aa09-42b98173d446","Type":"ContainerStarted","Data":"46e6a8009aec53edb56d30f12dd0a06431010682b5f0dc5dc30dd578a522bc1e"}
Nov 02 07:33:09.394999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:09.394671    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:09.436987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:09.436928    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s" podStartSLOduration=1.92627637 podStartE2EDuration="2.436915868s" totalImagesPullingTime="510.639498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:09.436342398 +0000 UTC m=+534.198779307" watchObservedRunningTime="2026-11-02 07:33:09.436915868 +0000 UTC m=+534.199352774"
Nov 02 07:33:12.368142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:12.368113    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 07:33:12.368552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:12.368257    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 07:33:17.721532 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:33:17.721488    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:33:20.401234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.401199    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-785d49bf89-46p4s"
Nov 02 07:33:20.471054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.471022    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:33:20.471271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.471247    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" podUID="4410f891-32d9-4658-a366-750e14bc4461" containerName="manager" containerID="cri-o://05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511" gracePeriod=30
Nov 02 07:33:20.711279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.711254    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:33:20.825152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.825115    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\" (UniqueName: \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\") pod \"4410f891-32d9-4658-a366-750e14bc4461\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") "
Nov 02 07:33:20.825362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.825279    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\") pod \"4410f891-32d9-4658-a366-750e14bc4461\" (UID: \"4410f891-32d9-4658-a366-750e14bc4461\") "
Nov 02 07:33:20.827164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.827135    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5" pod "4410f891-32d9-4658-a366-750e14bc4461" (UID: "4410f891-32d9-4658-a366-750e14bc4461"). InnerVolumeSpecName "kube-api-access-ddpv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:33:20.827164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.827155    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs" pod "4410f891-32d9-4658-a366-750e14bc4461" (UID: "4410f891-32d9-4658-a366-750e14bc4461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:33:20.926262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.926221    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddpv5\" (UniqueName: \"kubernetes.io/projected/4410f891-32d9-4658-a366-750e14bc4461-kube-api-access-ddpv5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:33:20.926262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:20.926255    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4410f891-32d9-4658-a366-750e14bc4461-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:33:21.443573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.443539    4515 generic.go:356] "Generic (PLEG): container finished" podID="4410f891-32d9-4658-a366-750e14bc4461" containerID="05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511" exitCode=0
Nov 02 07:33:21.443983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.443584    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb"
Nov 02 07:33:21.443983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.443612    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" event={"ID":"4410f891-32d9-4658-a366-750e14bc4461","Type":"ContainerDied","Data":"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"}
Nov 02 07:33:21.443983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.443651    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p2fzb" event={"ID":"4410f891-32d9-4658-a366-750e14bc4461","Type":"ContainerDied","Data":"960df09901fe1f83ec2b1104e777d612c14fb4d99a7e57e7f684f1446be9b17a"}
Nov 02 07:33:21.443983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.443672    4515 scope.go:122] "RemoveContainer" containerID="05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"
Nov 02 07:33:21.451828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.451810    4515 scope.go:122] "RemoveContainer" containerID="05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"
Nov 02 07:33:21.452093 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:33:21.452073    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511\": container with ID starting with 05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511 not found: ID does not exist" containerID="05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"
Nov 02 07:33:21.452174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.452105    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511"} err="failed to get container status \"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511\": rpc error: code = NotFound desc = could not find container \"05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511\": container with ID starting with 05984552f5669e10f175a9eec4acdcbd34e38918cbb08e21ded3a10721793511 not found: ID does not exist"
Nov 02 07:33:21.495108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.495076    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:33:21.507689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:21.507659    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p2fzb"]
Nov 02 07:33:23.370646 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:23.370605    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4410f891-32d9-4658-a366-750e14bc4461" path="/var/lib/kubelet/pods/4410f891-32d9-4658-a366-750e14bc4461/volumes"
Nov 02 07:33:27.574116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.574083    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"]
Nov 02 07:33:27.575064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.575036    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4410f891-32d9-4658-a366-750e14bc4461" containerName="manager"
Nov 02 07:33:27.575064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.575064    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4410f891-32d9-4658-a366-750e14bc4461" containerName="manager"
Nov 02 07:33:27.575246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.575176    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4410f891-32d9-4658-a366-750e14bc4461" containerName="manager"
Nov 02 07:33:27.582665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.582596    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.598150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.598126    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 07:33:27.598488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.598468    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"]
Nov 02 07:33:27.609205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.609176    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 07:33:27.609418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.609364    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dj6tl\""
Nov 02 07:33:27.609418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.609386    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 07:33:27.685719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685679    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.685719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685719    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.685929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685743    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cxm\" (UniqueName: \"kubernetes.io/projected/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-kube-api-access-69cxm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.685929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685867    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.686001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685934    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.686001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.685960    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.786704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.786662    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.786704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.786708    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.786955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.786760    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.786955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.786784    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.786955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.786819    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cxm\" (UniqueName: \"kubernetes.io/projected/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-kube-api-access-69cxm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.787114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.787006    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.787176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.787117    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.787176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.787157    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.787369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.787350    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.789303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.789272    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.789875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.789853    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.823616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.823588    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cxm\" (UniqueName: \"kubernetes.io/projected/d2c2ca8c-58ad-4e92-9d4b-caab320c14d5-kube-api-access-69cxm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2\" (UID: \"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:27.897169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:27.897079    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:28.193580 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:28.193550    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"]
Nov 02 07:33:28.193677 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:33:28.193599    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c2ca8c_58ad_4e92_9d4b_caab320c14d5.slice/crio-d4f989e63a4bbd0c7ac0536b08babd17aa097cb9bc5a0225b0e8204a91f22819 WatchSource:0}: Error finding container d4f989e63a4bbd0c7ac0536b08babd17aa097cb9bc5a0225b0e8204a91f22819: Status 404 returned error can't find the container with id d4f989e63a4bbd0c7ac0536b08babd17aa097cb9bc5a0225b0e8204a91f22819
Nov 02 07:33:28.473866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:28.473771    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2" event={"ID":"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5","Type":"ContainerStarted","Data":"d4f989e63a4bbd0c7ac0536b08babd17aa097cb9bc5a0225b0e8204a91f22819"}
Nov 02 07:33:29.372859 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:29.372832    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:33:29.480655 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:29.480616    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2" event={"ID":"d2c2ca8c-58ad-4e92-9d4b-caab320c14d5","Type":"ContainerStarted","Data":"7e0082603f8d023dc69d52327741e67c244cff84ff758c23b9a2ecaac739f843"}
Nov 02 07:33:29.529375 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:29.529324    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2" podStartSLOduration=1.352401369 podStartE2EDuration="2.529294744s" totalImagesPullingTime="1.176893375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:29.528727944 +0000 UTC m=+554.291164849" watchObservedRunningTime="2026-11-02 07:33:29.529294744 +0000 UTC m=+554.291731628"
Nov 02 07:33:29.897738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:29.897701    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:34.152169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:34.152137    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:33:37.292784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.292749    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:33:37.697741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.697667    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"]
Nov 02 07:33:37.701474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.701446    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.745082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.744995    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 07:33:37.746721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.746692    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"]
Nov 02 07:33:37.774151 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774120    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.774326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774165    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7c6a8a-a628-4320-9f42-726b3469c007-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.774326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774244    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.774326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774280    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vq4\" (UniqueName: \"kubernetes.io/projected/8e7c6a8a-a628-4320-9f42-726b3469c007-kube-api-access-l5vq4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.774326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774325    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.774466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.774379    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.874832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874796    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.874832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874833    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874879    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874909    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7c6a8a-a628-4320-9f42-726b3469c007-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.874964    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vq4\" (UniqueName: \"kubernetes.io/projected/8e7c6a8a-a628-4320-9f42-726b3469c007-kube-api-access-l5vq4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.875542    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.875553    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.875757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.875558    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.877460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.877438    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7c6a8a-a628-4320-9f42-726b3469c007-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.877557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.877529    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7c6a8a-a628-4320-9f42-726b3469c007-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.897564 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.897519    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:37.900355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.900331    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vq4\" (UniqueName: \"kubernetes.io/projected/8e7c6a8a-a628-4320-9f42-726b3469c007-kube-api-access-l5vq4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-glvhv\" (UID: \"8e7c6a8a-a628-4320-9f42-726b3469c007\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:37.910438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:37.910416    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:38.013630 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:38.013597    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:38.310770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:38.310741    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"]
Nov 02 07:33:38.311218 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:33:38.311157    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7c6a8a_a628_4320_9f42_726b3469c007.slice/crio-3263b7ce53377b9990661414983360a280bc778383106ad12ecec5d6e7b8c024 WatchSource:0}: Error finding container 3263b7ce53377b9990661414983360a280bc778383106ad12ecec5d6e7b8c024: Status 404 returned error can't find the container with id 3263b7ce53377b9990661414983360a280bc778383106ad12ecec5d6e7b8c024
Nov 02 07:33:38.518670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:38.518638    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv" event={"ID":"8e7c6a8a-a628-4320-9f42-726b3469c007","Type":"ContainerStarted","Data":"3263b7ce53377b9990661414983360a280bc778383106ad12ecec5d6e7b8c024"}
Nov 02 07:33:38.530671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:38.530646    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2"
Nov 02 07:33:39.524971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:39.524931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv" event={"ID":"8e7c6a8a-a628-4320-9f42-726b3469c007","Type":"ContainerStarted","Data":"419204753b052ad828ad32eea892dadfcdad9a0e14649ba41bd586e739e3884f"}
Nov 02 07:33:39.568497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:39.568444    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv" podStartSLOduration=2.389196396 podStartE2EDuration="2.56843111s" totalImagesPullingTime="179.234714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:39.567204685 +0000 UTC m=+564.329641593" watchObservedRunningTime="2026-11-02 07:33:39.56843111 +0000 UTC m=+564.330868017"
Nov 02 07:33:39.987027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:39.986953    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"]
Nov 02 07:33:39.990630 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:39.990607    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.013797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.013765    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:40.020112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.020087    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 07:33:40.021648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.021625    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"]
Nov 02 07:33:40.089809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.089774    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.089969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.089830    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.089969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.089925    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.089969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.089958    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2vf\" (UniqueName: \"kubernetes.io/projected/0724f5a8-427f-474a-8f84-74582d627536-kube-api-access-br2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.090114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.090013    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0724f5a8-427f-474a-8f84-74582d627536-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.090114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.090045    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190593    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190672    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190718    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190751    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190773    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2vf\" (UniqueName: \"kubernetes.io/projected/0724f5a8-427f-474a-8f84-74582d627536-kube-api-access-br2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.190992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.190808    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0724f5a8-427f-474a-8f84-74582d627536-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.191154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.191120    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.191292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.191154    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.191292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.191199    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.192873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.192852    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0724f5a8-427f-474a-8f84-74582d627536-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.193209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.193191    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0724f5a8-427f-474a-8f84-74582d627536-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.215227 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.215201    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2vf\" (UniqueName: \"kubernetes.io/projected/0724f5a8-427f-474a-8f84-74582d627536-kube-api-access-br2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6\" (UID: \"0724f5a8-427f-474a-8f84-74582d627536\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.304635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.304596    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:40.602728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:40.602699    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"]
Nov 02 07:33:40.603161 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:33:40.603139    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724f5a8_427f_474a_8f84_74582d627536.slice/crio-0b625e0b04ea66e0619769b275633bf473fa89087c81a722a1c3634e6049d910 WatchSource:0}: Error finding container 0b625e0b04ea66e0619769b275633bf473fa89087c81a722a1c3634e6049d910: Status 404 returned error can't find the container with id 0b625e0b04ea66e0619769b275633bf473fa89087c81a722a1c3634e6049d910
Nov 02 07:33:41.535360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:41.535297    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6" event={"ID":"0724f5a8-427f-474a-8f84-74582d627536","Type":"ContainerStarted","Data":"d57ecf10098d4569cfcdfdf7aa392baf1e0806dbe4b1b779e3ac583f930ad566"}
Nov 02 07:33:41.535642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:41.535383    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6" event={"ID":"0724f5a8-427f-474a-8f84-74582d627536","Type":"ContainerStarted","Data":"0b625e0b04ea66e0619769b275633bf473fa89087c81a722a1c3634e6049d910"}
Nov 02 07:33:41.579928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:41.579878    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6" podStartSLOduration=2.408552555 podStartE2EDuration="2.579863683s" totalImagesPullingTime="171.311128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:41.578723326 +0000 UTC m=+566.341160232" watchObservedRunningTime="2026-11-02 07:33:41.579863683 +0000 UTC m=+566.342300624"
Nov 02 07:33:41.726536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:41.726429    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:33:42.304875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:42.304832    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:43.721277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.721245    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xbgtq"]
Nov 02 07:33:43.724805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.724783    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.756534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.756502    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 07:33:43.757858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.757837    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xbgtq"]
Nov 02 07:33:43.816732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816700    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-home\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.816894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816743    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.816894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816763    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkw7w\" (UniqueName: \"kubernetes.io/projected/047a025e-3d5b-468c-ae1c-b11786ec507c-kube-api-access-zkw7w\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.816894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816882    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.816992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816913    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.816992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.816936    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047a025e-3d5b-468c-ae1c-b11786ec507c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.917592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917561    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-home\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.917786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917605    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.917786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917772    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkw7w\" (UniqueName: \"kubernetes.io/projected/047a025e-3d5b-468c-ae1c-b11786ec507c-kube-api-access-zkw7w\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.917920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917876    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.917920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917897    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.918027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917921    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047a025e-3d5b-468c-ae1c-b11786ec507c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.918027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.917996    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-home\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.918278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.918247    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.918417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.918258    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.919875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.919853    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047a025e-3d5b-468c-ae1c-b11786ec507c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.920427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.920407    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047a025e-3d5b-468c-ae1c-b11786ec507c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:43.941278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:43.941253    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkw7w\" (UniqueName: \"kubernetes.io/projected/047a025e-3d5b-468c-ae1c-b11786ec507c-kube-api-access-zkw7w\") pod \"echo-simulator-kserve-64b6f8f96c-xbgtq\" (UID: \"047a025e-3d5b-468c-ae1c-b11786ec507c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:44.038375 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:44.038336    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:44.336248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:44.336216    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xbgtq"]
Nov 02 07:33:44.336553 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:33:44.336526    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047a025e_3d5b_468c_ae1c_b11786ec507c.slice/crio-e8389a50c49f81f57662dbd11e4eedfe32e0004bc7fe65054f4135d52d464275 WatchSource:0}: Error finding container e8389a50c49f81f57662dbd11e4eedfe32e0004bc7fe65054f4135d52d464275: Status 404 returned error can't find the container with id e8389a50c49f81f57662dbd11e4eedfe32e0004bc7fe65054f4135d52d464275
Nov 02 07:33:44.550593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:44.550561    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq" event={"ID":"047a025e-3d5b-468c-ae1c-b11786ec507c","Type":"ContainerStarted","Data":"e8389a50c49f81f57662dbd11e4eedfe32e0004bc7fe65054f4135d52d464275"}
Nov 02 07:33:46.561537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:46.561492    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq" event={"ID":"047a025e-3d5b-468c-ae1c-b11786ec507c","Type":"ContainerStarted","Data":"958db0d8f9bcdd617778c1fcace0bbc68ac881fcc026ec34937c7354a89d0143"}
Nov 02 07:33:46.605048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:46.604994    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq" podStartSLOduration=2.223790376 podStartE2EDuration="3.604981891s" totalImagesPullingTime="1.381191515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:46.604178213 +0000 UTC m=+571.366615118" watchObservedRunningTime="2026-11-02 07:33:46.604981891 +0000 UTC m=+571.367418797"
Nov 02 07:33:47.039018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:47.038979    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:47.721427 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:33:47.721390    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:33:48.014259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:48.014216    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:48.026998 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:48.026966    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:48.536722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:48.536687    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:33:48.580362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:48.580328    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-glvhv"
Nov 02 07:33:50.304920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:50.304889    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:50.317285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:50.317253    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:50.588378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:50.588291    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6"
Nov 02 07:33:54.039402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:54.039364    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:54.051186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:54.051155    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:33:54.605704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:33:54.605669    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xbgtq"
Nov 02 07:34:02.506344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.506239    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 07:34:02.512762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.512740    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.530436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.530401    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7cxdc\""
Nov 02 07:34:02.544805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.544773    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 07:34:02.570045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.569998    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570060    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570243    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570396    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrnj\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570452    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570527    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570592    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570654    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.570775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.570680    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.671673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.671638    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.671852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.671689    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrnj\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.671852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.671710    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.671852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.671741    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.671981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.671924    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672010    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672047    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672068    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672078    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672120    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672153    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672359    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672548    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.672676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.672657    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.674088 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.674059    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.674426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.674410    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.695221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.695194    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.695355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.695339    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrnj\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hm547\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:02.826693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:02.826611    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:03.152724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:03.152691    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 07:34:03.152724 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:03.152714    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf1506e_87b5_4951_a2fc_64b71d997a0b.slice/crio-8021217daf34cbc4bbc59b991a4e0108a9a445a4d54d2ee000375529d9069d30 WatchSource:0}: Error finding container 8021217daf34cbc4bbc59b991a4e0108a9a445a4d54d2ee000375529d9069d30: Status 404 returned error can't find the container with id 8021217daf34cbc4bbc59b991a4e0108a9a445a4d54d2ee000375529d9069d30
Nov 02 07:34:03.631944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:03.631913    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" event={"ID":"daf1506e-87b5-4951-a2fc-64b71d997a0b","Type":"ContainerStarted","Data":"8021217daf34cbc4bbc59b991a4e0108a9a445a4d54d2ee000375529d9069d30"}
Nov 02 07:34:06.000267 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.000231    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:06.000680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.000325    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:06.000680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.000376    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:06.653450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.653408    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" event={"ID":"daf1506e-87b5-4951-a2fc-64b71d997a0b","Type":"ContainerStarted","Data":"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"}
Nov 02 07:34:06.727785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.727719    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podStartSLOduration=1.882904061 podStartE2EDuration="4.727688061s" totalImagesPullingTime="2.844784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:06.712486853 +0000 UTC m=+591.474923759" watchObservedRunningTime="2026-11-02 07:34:06.727688061 +0000 UTC m=+591.490124970"
Nov 02 07:34:06.830478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:06.830437    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:07.834336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:07.834272    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:07.835000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:07.834965    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:08.841483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:08.841442    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:08.841898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:08.841501    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:09.839334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:09.839164    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:09.839334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:09.839222    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:10.835795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:10.835758    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:10.836418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:10.836385    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:11.843417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:11.842881    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:11.843417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:11.842941    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:12.827834 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:12.827792    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 07:34:12.828139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:12.827857    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 07:34:14.256158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:14.256109    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:14.256888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:14.256860    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:14.257694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:14.257669    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 07:34:17.543265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.543231    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 07:34:17.552892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.552862    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.582232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.582203    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 07:34:17.722922 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:17.722889    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:34:17.734453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734419    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734462    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734546    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734575    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734592    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734612    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734754    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734798    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.734844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.734815    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836220    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836336    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836362    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836416    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836443    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836477    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836500    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836521    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.836962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836904    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.837013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.837013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.836965    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.837223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.837201    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.837329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.837231    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.837329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.837237    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.838724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.838703    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.839386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.839370    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.860340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.860286    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.860472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.860450    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8fmqz\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:17.868213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:17.868194    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:18.180103 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:18.180069    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e197b26_0b5d_40c3_b4ff_7f120cec07d9.slice/crio-1196f46a786827984594d2ec7724546f328c8041e9854b26f11aee011f81f3d7 WatchSource:0}: Error finding container 1196f46a786827984594d2ec7724546f328c8041e9854b26f11aee011f81f3d7: Status 404 returned error can't find the container with id 1196f46a786827984594d2ec7724546f328c8041e9854b26f11aee011f81f3d7
Nov 02 07:34:18.180103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.180091    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 07:34:18.182239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.182206    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.182340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.182272    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.182340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.182303    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.706882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.706842    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" event={"ID":"1e197b26-0b5d-40c3-b4ff-7f120cec07d9","Type":"ContainerStarted","Data":"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"}
Nov 02 07:34:18.707296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.706922    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" event={"ID":"1e197b26-0b5d-40c3-b4ff-7f120cec07d9","Type":"ContainerStarted","Data":"1196f46a786827984594d2ec7724546f328c8041e9854b26f11aee011f81f3d7"}
Nov 02 07:34:18.758815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.758756    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podStartSLOduration=1.758741633 podStartE2EDuration="1.758741633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:18.757180576 +0000 UTC m=+603.519617583" watchObservedRunningTime="2026-11-02 07:34:18.758741633 +0000 UTC m=+603.521178539"
Nov 02 07:34:18.868881 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:18.868846    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:19.870629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:19.870594    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:19.871019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:19.870645    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:20.868765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:20.868730    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:20.868977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:20.868781    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:21.869612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:21.869580    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:21.870021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:21.869635    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:22.868712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:22.868676    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:22.868905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:22.868745    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:23.868938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:23.868898    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:23.869384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:23.868961    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:24.869575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:24.869527    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 07:34:24.870005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:24.869600    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 07:34:26.531061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:26.531020    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:26.531833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:26.531427    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:26.532221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:26.532203    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 07:34:31.236358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.236321    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:34:31.239829 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.239812    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.259264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.259240    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-935d4e21-openshift-default-dockercfg-clvk2\""
Nov 02 07:34:31.260602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260577    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260624    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260683    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260701    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260798    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260844    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260867    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5652\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.260897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260890    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.261119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.260910    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.273328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.273278    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:34:31.361788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361757    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.361788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361794    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361813    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361853    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361882    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361901    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5652\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.361917    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362134    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362213    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362301    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362373    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362447    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362522    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.362743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.362718    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.364283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.364257    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.364390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.364379    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.388743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.388712    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5652\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.392210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.392189    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\") pod \"e2e-ait-935d4e21-openshift-default-555f96899-bfw85\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") " pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.554071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.554031    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:31.885232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.885202    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:34:31.886909 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:31.886878    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da38b4_f74b_4794_b57e_38ea99cea1e0.slice/crio-6339e76a712e745d033445c16c386e324357d47a8f8937c633d0bb7a180feb68 WatchSource:0}: Error finding container 6339e76a712e745d033445c16c386e324357d47a8f8937c633d0bb7a180feb68: Status 404 returned error can't find the container with id 6339e76a712e745d033445c16c386e324357d47a8f8937c633d0bb7a180feb68
Nov 02 07:34:31.889291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.889256    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:31.889414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.889391    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:31.889485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:31.889447    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.793069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.793031    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerStarted","Data":"584688a1e76f97355e1e1e4ef95c1ac556a11b641fb2d46c38082ba82d627549"}
Nov 02 07:34:32.793592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.793092    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerStarted","Data":"6339e76a712e745d033445c16c386e324357d47a8f8937c633d0bb7a180feb68"}
Nov 02 07:34:32.796816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.796775    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:34:32.801049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.801023    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.819057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.819032    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8ff8e809-openshift-default-dockercfg-5b8zp\""
Nov 02 07:34:32.828261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.828231    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:34:32.846855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.846806    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podStartSLOduration=1.846793104 podStartE2EDuration="1.846793104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:32.84573387 +0000 UTC m=+617.608170775" watchObservedRunningTime="2026-11-02 07:34:32.846793104 +0000 UTC m=+617.609230009"
Nov 02 07:34:32.877384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877354    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877395    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877488    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877522    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlj9\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877700    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877743    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877881 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877864    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877913    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.877974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.877936    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.979571 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.979533    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.979765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.979589    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.979765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.979626    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.979765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.979655    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlj9\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.980055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.980015    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.980579 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.980217    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.980824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.980107    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.980824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.980554    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.980824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.980798    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.981053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.981033    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.981192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.981174    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.981337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.981284    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.981500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.981477    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.981699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.981678    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.982382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.982362    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:32.982605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:32.982583    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:33.006072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.006043    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlj9\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:33.006391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.006146    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\") pod \"e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") " pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:33.116550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.116450    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:33.432714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.432687    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:34:33.433189 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:33.433160    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c67379_ce1d_4477_ab19_fb01f8359db0.slice/crio-6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4 WatchSource:0}: Error finding container 6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4: Status 404 returned error can't find the container with id 6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4
Nov 02 07:34:33.435544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.435515    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:33.435606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.435578    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:33.435643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.435610    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:33.554557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.554519    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:33.560847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.560816    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:33.799902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.799864    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerStarted","Data":"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"}
Nov 02 07:34:33.800357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.799925    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerStarted","Data":"6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4"}
Nov 02 07:34:33.800357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.800130    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:33.801047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.801026    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:34:33.848485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:33.848421    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podStartSLOduration=1.848403319 podStartE2EDuration="1.848403319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:33.846377592 +0000 UTC m=+618.608814501" watchObservedRunningTime="2026-11-02 07:34:33.848403319 +0000 UTC m=+618.610840228"
Nov 02 07:34:34.117428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:34.117340    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:34.123006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:34.122978    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:34.804699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:34.804666    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:34.805547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:34.805527    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:36.140844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.140807    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:34:36.144389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.144365    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.162525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.162502    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b9780053\""
Nov 02 07:34:36.174388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.174356    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:34:36.212353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.210733    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b9780053-dockercfg-hkbvf\""
Nov 02 07:34:36.214932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.214896    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.215398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.215359    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnb2\" (UniqueName: \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.223976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.223944    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:34:36.228560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.228535    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.256130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.256083    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:34:36.316973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.316934    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.317181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.316993    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnb2\" (UniqueName: \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.317181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.317050    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.317181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.317103    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdv6m\" (UniqueName: \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.317794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.317767    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.345987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.345954    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnb2\" (UniqueName: \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\") pod \"payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.418254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.418176    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.418254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.418224    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdv6m\" (UniqueName: \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.418932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.418912    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.444043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.444015    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdv6m\" (UniqueName: \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\") pod \"payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.456900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.456873    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:36.552524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.552492    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:36.774400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.774372    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:34:36.774637 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:36.774612    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf847af58_af1f_418f_8969_8cefd1889729.slice/crio-bc9b6613ff9162dfe696d0a12cf929f58caeaaeb76c20b4aad9b568837366812 WatchSource:0}: Error finding container bc9b6613ff9162dfe696d0a12cf929f58caeaaeb76c20b4aad9b568837366812: Status 404 returned error can't find the container with id bc9b6613ff9162dfe696d0a12cf929f58caeaaeb76c20b4aad9b568837366812
Nov 02 07:34:36.816000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.815969    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" event={"ID":"f847af58-af1f-418f-8969-8cefd1889729","Type":"ContainerStarted","Data":"bc9b6613ff9162dfe696d0a12cf929f58caeaaeb76c20b4aad9b568837366812"}
Nov 02 07:34:36.862077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:36.862046    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:34:36.862217 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:36.862197    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749f6c1a_2475_4606_a5f0_4b297606ef71.slice/crio-e1b237400386e99b6034fcc4b231402744ba830055c0943612c7cf6d01592aa8 WatchSource:0}: Error finding container e1b237400386e99b6034fcc4b231402744ba830055c0943612c7cf6d01592aa8: Status 404 returned error can't find the container with id e1b237400386e99b6034fcc4b231402744ba830055c0943612c7cf6d01592aa8
Nov 02 07:34:37.821992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.821951    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" event={"ID":"f847af58-af1f-418f-8969-8cefd1889729","Type":"ContainerStarted","Data":"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"}
Nov 02 07:34:37.822493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.822020    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:37.823581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.823552    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" event={"ID":"749f6c1a-2475-4606-a5f0-4b297606ef71","Type":"ContainerStarted","Data":"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"}
Nov 02 07:34:37.823706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.823604    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" event={"ID":"749f6c1a-2475-4606-a5f0-4b297606ef71","Type":"ContainerStarted","Data":"e1b237400386e99b6034fcc4b231402744ba830055c0943612c7cf6d01592aa8"}
Nov 02 07:34:37.823706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.823636    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:37.881703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.881662    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:34:37.888346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.886173    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:37.910982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.910920    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" podStartSLOduration=1.9109017480000001 podStartE2EDuration="1.910901748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:37.909746344 +0000 UTC m=+622.672183252" watchObservedRunningTime="2026-11-02 07:34:37.910901748 +0000 UTC m=+622.673338655"
Nov 02 07:34:37.927941 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.927904    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-935d4e21\""
Nov 02 07:34:37.929146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.929115    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-935d4e21-dockercfg-c9cmh\""
Nov 02 07:34:37.934528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.934501    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:34:37.940458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.940422    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznj8\" (UniqueName: \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:37.940595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.940530    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:37.971073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.971014    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" podStartSLOduration=1.971000453 podStartE2EDuration="1.971000453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:37.970210654 +0000 UTC m=+622.732647560" watchObservedRunningTime="2026-11-02 07:34:37.971000453 +0000 UTC m=+622.733437359"
Nov 02 07:34:37.985722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.985685    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:34:37.992299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:37.992270    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.020467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.017095    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:34:38.027507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.027478    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b9780053-serving-cert\""
Nov 02 07:34:38.042189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042158    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.042371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042201    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fznj8\" (UniqueName: \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.042371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042244    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvc8\" (UniqueName: \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.042531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042510    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.042608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042593    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.042682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.042669    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.043256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.043237    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.070734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.070706    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznj8\" (UniqueName: \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\") pod \"payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.144109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.144013    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.144109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.144093    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.144367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.144139    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.144367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.144195    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvc8\" (UniqueName: \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.144367 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:38.144230    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-serving-cert: secret "maas-api-e2e-migrate-b9780053-serving-cert" not found
Nov 02 07:34:38.144367 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:38.144325    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls podName:e6840fad-7fd6-4581-9cbd-626954c81911 nodeName:}" failed. No retries permitted until 2026-11-02 07:34:38.644287147 +0000 UTC m=+623.406724032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls") pod "maas-api-e2e-migrate-b9780053-78d887b485-s52j4" (UID: "e6840fad-7fd6-4581-9cbd-626954c81911") : secret "maas-api-e2e-migrate-b9780053-serving-cert" not found
Nov 02 07:34:38.144750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.144732    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.146656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.146638    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.185698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.185662    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvc8\" (UniqueName: \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.220455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.220426    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.526300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.526272    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:34:38.526875 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:38.526836    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ceed76_f654_476c_99b4_36833ed6ecaa.slice/crio-064cd6e3e2244a66fe89ba5cab8c9e2c5c4c4910776c29c553bd55b61bb61615 WatchSource:0}: Error finding container 064cd6e3e2244a66fe89ba5cab8c9e2c5c4c4910776c29c553bd55b61bb61615: Status 404 returned error can't find the container with id 064cd6e3e2244a66fe89ba5cab8c9e2c5c4c4910776c29c553bd55b61bb61615
Nov 02 07:34:38.648681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.648644    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.651032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.651010    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") pod \"maas-api-e2e-migrate-b9780053-78d887b485-s52j4\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:38.830003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.829909    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" event={"ID":"f9ceed76-f654-476c-99b4-36833ed6ecaa","Type":"ContainerStarted","Data":"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"}
Nov 02 07:34:38.830500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.830275    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:38.830500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.830334    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" event={"ID":"f9ceed76-f654-476c-99b4-36833ed6ecaa","Type":"ContainerStarted","Data":"064cd6e3e2244a66fe89ba5cab8c9e2c5c4c4910776c29c553bd55b61bb61615"}
Nov 02 07:34:38.873795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.873742    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" podStartSLOduration=1.873730356 podStartE2EDuration="1.873730356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:38.87341886 +0000 UTC m=+623.635855796" watchObservedRunningTime="2026-11-02 07:34:38.873730356 +0000 UTC m=+623.636167261"
Nov 02 07:34:38.912399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:38.912362    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:39.219222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.219184    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:34:39.219361 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:39.219278    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6840fad_7fd6_4581_9cbd_626954c81911.slice/crio-97723a43368e3638f4581908f7d5173e9cc4ec1bc8c2b31537647b615d840408 WatchSource:0}: Error finding container 97723a43368e3638f4581908f7d5173e9cc4ec1bc8c2b31537647b615d840408: Status 404 returned error can't find the container with id 97723a43368e3638f4581908f7d5173e9cc4ec1bc8c2b31537647b615d840408
Nov 02 07:34:39.739013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.738983    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:34:39.742454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.742429    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.757383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.757363    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8ff8e809-serving-cert\""
Nov 02 07:34:39.763484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.763404    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:34:39.838144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.838107    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" event={"ID":"e6840fad-7fd6-4581-9cbd-626954c81911","Type":"ContainerStarted","Data":"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"}
Nov 02 07:34:39.838653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.838161    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" event={"ID":"e6840fad-7fd6-4581-9cbd-626954c81911","Type":"ContainerStarted","Data":"97723a43368e3638f4581908f7d5173e9cc4ec1bc8c2b31537647b615d840408"}
Nov 02 07:34:39.864455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.864417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.864640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.864493    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7nx\" (UniqueName: \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.864640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.864562    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.864640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.864605    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.881106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.881044    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" podStartSLOduration=2.450761129 podStartE2EDuration="2.881024161s" totalImagesPullingTime="430.263032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:39.879756031 +0000 UTC m=+624.642192942" watchObservedRunningTime="2026-11-02 07:34:39.881024161 +0000 UTC m=+624.643461067"
Nov 02 07:34:39.966179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.966097    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.966364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.966268    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7nx\" (UniqueName: \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.966364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.966298    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.966507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.966373    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.967346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.967301    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.969467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.969348    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.969626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.969512    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:39.990071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:39.990038    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7nx\" (UniqueName: \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\") pod \"maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:40.055951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.055914    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:40.358396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.358359    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:34:40.358626 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:40.358605    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f9e120_ed62_40f6_9407_101232be2e4e.slice/crio-075d78abf074697596294d81585bbae5260bdfe7a13135ca7625a5f73f36ac85 WatchSource:0}: Error finding container 075d78abf074697596294d81585bbae5260bdfe7a13135ca7625a5f73f36ac85: Status 404 returned error can't find the container with id 075d78abf074697596294d81585bbae5260bdfe7a13135ca7625a5f73f36ac85
Nov 02 07:34:40.359942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.359924    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:34:40.845522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.845479    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" event={"ID":"f4f9e120-ed62-40f6-9407-101232be2e4e","Type":"ContainerStarted","Data":"b6de1f68a0f87b46d78a2497f48af719f8b27df1722b0c211982153d1d2f52c4"}
Nov 02 07:34:40.845955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.845573    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" event={"ID":"f4f9e120-ed62-40f6-9407-101232be2e4e","Type":"ContainerStarted","Data":"075d78abf074697596294d81585bbae5260bdfe7a13135ca7625a5f73f36ac85"}
Nov 02 07:34:40.845955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.845857    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:40.845955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.845898    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:40.890471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:40.890415    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" podStartSLOduration=1.567420288 podStartE2EDuration="1.890400419s" totalImagesPullingTime="322.980131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:40.88817699 +0000 UTC m=+625.650613896" watchObservedRunningTime="2026-11-02 07:34:40.890400419 +0000 UTC m=+625.652837325"
Nov 02 07:34:42.246193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:42.246161    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:34:45.441330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.440634    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:34:45.472002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.471846    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:34:45.472002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.471947    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.487230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.487201    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8c43c08e\""
Nov 02 07:34:45.499796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.499762    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8c43c08e-dockercfg-4jscs\""
Nov 02 07:34:45.662561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.662523    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.662731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.662612    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lgkg\" (UniqueName: \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.763267 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.763234    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgkg\" (UniqueName: \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.763486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.763339    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.764056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.764035    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:45.788422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:45.788384    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgkg\" (UniqueName: \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\") pod \"payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") " pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:46.056613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.056523    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:34:46.060681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.060659    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.075714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.075687    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-be1fb7c9-serving-cert\""
Nov 02 07:34:46.088208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.088165    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:46.092777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.092744    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:34:46.166324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.166276    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.166489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.166361    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7px\" (UniqueName: \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.166489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.166417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.166489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.166436    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.267447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.267414    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.267615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.267494    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7px\" (UniqueName: \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.267615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.267562    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.267615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.267587    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.268346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.268292    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.270080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.270057    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.270222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.270207    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.293331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.292983    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7px\" (UniqueName: \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\") pod \"maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.374005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.373922    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:46.430776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.430741    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:34:46.700011 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:46.699964    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc028b96_6567_4257_9a75_67b6a4e4313b.slice/crio-06bd694f632d9d7b832f47cb79bff07b70e0b156fb676a7b49e35282a8d28530 WatchSource:0}: Error finding container 06bd694f632d9d7b832f47cb79bff07b70e0b156fb676a7b49e35282a8d28530: Status 404 returned error can't find the container with id 06bd694f632d9d7b832f47cb79bff07b70e0b156fb676a7b49e35282a8d28530
Nov 02 07:34:46.700633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.700611    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:34:46.855869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.855840    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:34:46.856302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.856278    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:34:46.875851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.875814    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" event={"ID":"cc028b96-6567-4257-9a75-67b6a4e4313b","Type":"ContainerStarted","Data":"06bd694f632d9d7b832f47cb79bff07b70e0b156fb676a7b49e35282a8d28530"}
Nov 02 07:34:46.877538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.877512    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" event={"ID":"6087761a-ce01-47e1-9203-0d9848d28892","Type":"ContainerStarted","Data":"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"}
Nov 02 07:34:46.877650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.877554    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" event={"ID":"6087761a-ce01-47e1-9203-0d9848d28892","Type":"ContainerStarted","Data":"e1996cb4e709f719f35e47563cee315430c5dc2de22891d5df4318a17742ab66"}
Nov 02 07:34:46.877650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.877584    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:46.989500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:46.989387    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" podStartSLOduration=1.98936686 podStartE2EDuration="1.98936686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:46.987768867 +0000 UTC m=+631.750205772" watchObservedRunningTime="2026-11-02 07:34:46.98936686 +0000 UTC m=+631.751803768"
Nov 02 07:34:47.766484 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:47.766085    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:34:47.884406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:47.884355    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" event={"ID":"cc028b96-6567-4257-9a75-67b6a4e4313b","Type":"ContainerStarted","Data":"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"}
Nov 02 07:34:47.884620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:47.884471    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:47.938333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:47.930495    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" podStartSLOduration=1.5174569070000001 podStartE2EDuration="1.930476029s" totalImagesPullingTime="413.019122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:47.929092151 +0000 UTC m=+632.691529060" watchObservedRunningTime="2026-11-02 07:34:47.930476029 +0000 UTC m=+632.692912937"
Nov 02 07:34:48.842131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:48.842087    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:34:48.842684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:48.842171    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:34:49.847672 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:49.847629    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:34:50.847498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:50.847462    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:34:50.847685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:50.847552    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:34:51.857116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.857083    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:34:51.861896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.861874    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:51.894945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.894913    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8c43c08e-serving-cert\""
Nov 02 07:34:51.895708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.895683    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:34:51.957080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.957039    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87s6j\" (UniqueName: \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:51.957246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.957211    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:51.957293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.957252    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:51.957293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:51.957282    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.058434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.058397    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.059268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.058818    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.059753 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:52.058744    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-serving-cert: secret "maas-api-e2e-derive-8c43c08e-serving-cert" not found
Nov 02 07:34:52.059753 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:34:52.059535    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls podName:2cf22a34-6257-4d2d-862a-a194ce0b32f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:34:52.559511951 +0000 UTC m=+637.321948850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls") pod "maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" (UID: "2cf22a34-6257-4d2d-862a-a194ce0b32f8") : secret "maas-api-e2e-derive-8c43c08e-serving-cert" not found
Nov 02 07:34:52.060688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.060644    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.060818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.060800    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.061016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.061002    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s6j\" (UniqueName: \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.063415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.063393    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.084985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.084955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87s6j\" (UniqueName: \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.566816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.566770    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.569201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.569169    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") pod \"maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:52.776286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:52.774630    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:53.105743 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:34:53.105704    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf22a34_6257_4d2d_862a_a194ce0b32f8.slice/crio-11af97383ea637a3d9e284a620b3318cb6e153b20b980f65b9b5cb1ed07deb76 WatchSource:0}: Error finding container 11af97383ea637a3d9e284a620b3318cb6e153b20b980f65b9b5cb1ed07deb76: Status 404 returned error can't find the container with id 11af97383ea637a3d9e284a620b3318cb6e153b20b980f65b9b5cb1ed07deb76
Nov 02 07:34:53.106345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.106295    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:34:53.344335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.327901    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:34:53.903653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.903622    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:34:53.960772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.960741    4515 generic.go:356] "Generic (PLEG): container finished" podID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerID="2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17" exitCode=0
Nov 02 07:34:53.960960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.960817    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerDied","Data":"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"}
Nov 02 07:34:53.961352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.961297    4515 scope.go:122] "RemoveContainer" containerID="2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"
Nov 02 07:34:53.962810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.962688    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:53.962905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.962823    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:53.962905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.962873    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:53.963752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.963729    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" event={"ID":"2cf22a34-6257-4d2d-862a-a194ce0b32f8","Type":"ContainerStarted","Data":"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"}
Nov 02 07:34:53.963820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.963775    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" event={"ID":"2cf22a34-6257-4d2d-862a-a194ce0b32f8","Type":"ContainerStarted","Data":"11af97383ea637a3d9e284a620b3318cb6e153b20b980f65b9b5cb1ed07deb76"}
Nov 02 07:34:53.964224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:53.964208    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:34:54.059335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:54.056794    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" podStartSLOduration=2.68917873 podStartE2EDuration="3.056775698s" totalImagesPullingTime="367.596968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:54.056196139 +0000 UTC m=+638.818633045" watchObservedRunningTime="2026-11-02 07:34:54.056775698 +0000 UTC m=+638.819212609"
Nov 02 07:34:54.972389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:54.972348    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerStarted","Data":"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"}
Nov 02 07:34:55.127754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:55.127715    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:34:55.128116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:55.128092    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:34:55.133343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:55.128266    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:34:56.122154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:56.121699    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:34:56.122154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:56.121766    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:34:57.125335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:57.122745    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:34:57.125335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:57.122842    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:34:57.902667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:57.901628    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:34:58.120847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.120811    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:34:58.121168 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.121117    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:34:58.994926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.994838    4515 generic.go:356] "Generic (PLEG): container finished" podID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerID="584688a1e76f97355e1e1e4ef95c1ac556a11b641fb2d46c38082ba82d627549" exitCode=0
Nov 02 07:34:58.995372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.994915    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerDied","Data":"584688a1e76f97355e1e1e4ef95c1ac556a11b641fb2d46c38082ba82d627549"}
Nov 02 07:34:58.995372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.995282    4515 scope.go:122] "RemoveContainer" containerID="584688a1e76f97355e1e1e4ef95c1ac556a11b641fb2d46c38082ba82d627549"
Nov 02 07:34:58.996543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.996513    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:58.996624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.996582    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:58.996624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:58.996617    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:59.141333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:59.137130    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:34:59.141333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:34:59.137200    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:35:00.001963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.001924    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerStarted","Data":"1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d"}
Nov 02 07:35:00.120337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.119976    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 07:35:00.120337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.120051    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 07:35:00.555840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.555807    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:00.557707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.556500    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 07:35:00.557707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.556578    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 07:35:00.984704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:00.984621    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:35:01.148432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:01.148268    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:35:01.149376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:01.149353    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:35:01.150204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:01.150185    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:35:01.555466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:01.555430    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 07:35:01.555649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:01.555493    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 07:35:02.554822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:02.554783    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 07:35:02.555215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:02.554840    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 07:35:03.555114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:03.555072    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 07:35:03.555713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:03.555147    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 07:35:03.743693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:03.743656    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:03.800527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:03.800487    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:04.554903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:04.554867    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 07:35:04.555080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:04.554920    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 07:35:05.183621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:05.183588    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:35:05.221024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:05.220988    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:35:05.559216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:05.559183    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:05.560341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:05.560297    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:10.185090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.185010    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" podUID="749f6c1a-2475-4606-a5f0-4b297606ef71" containerName="payload-processing" containerID="cri-o://7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea" gracePeriod=25
Nov 02 07:35:10.221892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.221832    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" podUID="f847af58-af1f-418f-8969-8cefd1889729" containerName="payload-pre-processing" containerID="cri-o://b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0" gracePeriod=25
Nov 02 07:35:10.486481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.486421    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:35:10.503839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.503813    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:35:10.614503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.614462    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\") pod \"f847af58-af1f-418f-8969-8cefd1889729\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") "
Nov 02 07:35:10.614693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.614534    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\" (UniqueName: \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\") pod \"749f6c1a-2475-4606-a5f0-4b297606ef71\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") "
Nov 02 07:35:10.614693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.614657    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\" (UniqueName: \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\") pod \"f847af58-af1f-418f-8969-8cefd1889729\" (UID: \"f847af58-af1f-418f-8969-8cefd1889729\") "
Nov 02 07:35:10.614810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.614693    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\") pod \"749f6c1a-2475-4606-a5f0-4b297606ef71\" (UID: \"749f6c1a-2475-4606-a5f0-4b297606ef71\") "
Nov 02 07:35:10.614857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.614833    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume" pod "f847af58-af1f-418f-8969-8cefd1889729" (UID: "f847af58-af1f-418f-8969-8cefd1889729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:10.615039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.615020    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f847af58-af1f-418f-8969-8cefd1889729-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:10.615187 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.615166    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume" pod "749f6c1a-2475-4606-a5f0-4b297606ef71" (UID: "749f6c1a-2475-4606-a5f0-4b297606ef71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:10.616831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.616806    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m" pod "749f6c1a-2475-4606-a5f0-4b297606ef71" (UID: "749f6c1a-2475-4606-a5f0-4b297606ef71"). InnerVolumeSpecName "kube-api-access-xdv6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:10.616923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.616882    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2" pod "f847af58-af1f-418f-8969-8cefd1889729" (UID: "f847af58-af1f-418f-8969-8cefd1889729"). InnerVolumeSpecName "kube-api-access-wlnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:10.716155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.716071    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdv6m\" (UniqueName: \"kubernetes.io/projected/749f6c1a-2475-4606-a5f0-4b297606ef71-kube-api-access-xdv6m\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:10.716155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.716101    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlnb2\" (UniqueName: \"kubernetes.io/projected/f847af58-af1f-418f-8969-8cefd1889729-kube-api-access-wlnb2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:10.716155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:10.716114    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749f6c1a-2475-4606-a5f0-4b297606ef71-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.043916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.043854    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:35:11.077534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.077502    4515 generic.go:356] "Generic (PLEG): container finished" podID="f847af58-af1f-418f-8969-8cefd1889729" containerID="b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0" exitCode=0
Nov 02 07:35:11.077721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.077549    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"
Nov 02 07:35:11.077721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.077590    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" event={"ID":"f847af58-af1f-418f-8969-8cefd1889729","Type":"ContainerDied","Data":"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"}
Nov 02 07:35:11.077721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.077649    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl" event={"ID":"f847af58-af1f-418f-8969-8cefd1889729","Type":"ContainerDied","Data":"bc9b6613ff9162dfe696d0a12cf929f58caeaaeb76c20b4aad9b568837366812"}
Nov 02 07:35:11.077721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.077672    4515 scope.go:122] "RemoveContainer" containerID="b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"
Nov 02 07:35:11.079054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.079025    4515 generic.go:356] "Generic (PLEG): container finished" podID="749f6c1a-2475-4606-a5f0-4b297606ef71" containerID="7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea" exitCode=0
Nov 02 07:35:11.079149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.079066    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" event={"ID":"749f6c1a-2475-4606-a5f0-4b297606ef71","Type":"ContainerDied","Data":"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"}
Nov 02 07:35:11.079149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.079068    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"
Nov 02 07:35:11.079149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.079103    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t" event={"ID":"749f6c1a-2475-4606-a5f0-4b297606ef71","Type":"ContainerDied","Data":"e1b237400386e99b6034fcc4b231402744ba830055c0943612c7cf6d01592aa8"}
Nov 02 07:35:11.090588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.090497    4515 scope.go:122] "RemoveContainer" containerID="b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"
Nov 02 07:35:11.091162 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:11.091135    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0\": container with ID starting with b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0 not found: ID does not exist" containerID="b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"
Nov 02 07:35:11.091415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.091359    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0"} err="failed to get container status \"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0\": rpc error: code = NotFound desc = could not find container \"b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0\": container with ID starting with b91f5f210b4107ad24ec6e740679d5060eb78f47e530bf3e3a5439f5d3193af0 not found: ID does not exist"
Nov 02 07:35:11.091415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.091413    4515 scope.go:122] "RemoveContainer" containerID="7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"
Nov 02 07:35:11.102439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.102346    4515 scope.go:122] "RemoveContainer" containerID="7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"
Nov 02 07:35:11.102712 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:11.102681    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea\": container with ID starting with 7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea not found: ID does not exist" containerID="7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"
Nov 02 07:35:11.102807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.102721    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea"} err="failed to get container status \"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea\": rpc error: code = NotFound desc = could not find container \"7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea\": container with ID starting with 7a793e452f1c635dc41d7a8803faf5def4e10a915850de701e71db45cd4f23ea not found: ID does not exist"
Nov 02 07:35:11.154003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.153920    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:35:11.158087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.158059    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9780053-755fd4dd7f-2vdwl"]
Nov 02 07:35:11.240472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.240436    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:35:11.262043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.262007    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9780053-68d6b99f49-zqg9t"]
Nov 02 07:35:11.371846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.371762    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749f6c1a-2475-4606-a5f0-4b297606ef71" path="/var/lib/kubelet/pods/749f6c1a-2475-4606-a5f0-4b297606ef71/volumes"
Nov 02 07:35:11.372145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.372131    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f847af58-af1f-418f-8969-8cefd1889729" path="/var/lib/kubelet/pods/f847af58-af1f-418f-8969-8cefd1889729/volumes"
Nov 02 07:35:11.454002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.453974    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:35:11.454189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.454165    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" podUID="e6840fad-7fd6-4581-9cbd-626954c81911" containerName="maas-api" containerID="cri-o://8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b" gracePeriod=30
Nov 02 07:35:11.710217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.710185    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:35:11.831173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.831138    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\") pod \"e6840fad-7fd6-4581-9cbd-626954c81911\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") "
Nov 02 07:35:11.831403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.831198    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") pod \"e6840fad-7fd6-4581-9cbd-626954c81911\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") "
Nov 02 07:35:11.831403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.831233    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\") pod \"e6840fad-7fd6-4581-9cbd-626954c81911\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") "
Nov 02 07:35:11.831403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.831354    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\" (UniqueName: \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\") pod \"e6840fad-7fd6-4581-9cbd-626954c81911\" (UID: \"e6840fad-7fd6-4581-9cbd-626954c81911\") "
Nov 02 07:35:11.831741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.831709    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca" pod "e6840fad-7fd6-4581-9cbd-626954c81911" (UID: "e6840fad-7fd6-4581-9cbd-626954c81911"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:11.833569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.833538    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs" pod "e6840fad-7fd6-4581-9cbd-626954c81911" (UID: "e6840fad-7fd6-4581-9cbd-626954c81911"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:11.833569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.833549    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8" pod "e6840fad-7fd6-4581-9cbd-626954c81911" (UID: "e6840fad-7fd6-4581-9cbd-626954c81911"). InnerVolumeSpecName "kube-api-access-tsvc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:11.833769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.833641    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls" pod "e6840fad-7fd6-4581-9cbd-626954c81911" (UID: "e6840fad-7fd6-4581-9cbd-626954c81911"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:11.854562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.854525    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:35:11.854806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.854776    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" podUID="f4f9e120-ed62-40f6-9407-101232be2e4e" containerName="maas-api" containerID="cri-o://b6de1f68a0f87b46d78a2497f48af719f8b27df1722b0c211982153d1d2f52c4" gracePeriod=30
Nov 02 07:35:11.933131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.933042    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.933131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.933083    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6840fad-7fd6-4581-9cbd-626954c81911-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.933131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.933099    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6840fad-7fd6-4581-9cbd-626954c81911-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.933131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:11.933114    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvc8\" (UniqueName: \"kubernetes.io/projected/e6840fad-7fd6-4581-9cbd-626954c81911-kube-api-access-tsvc8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.087285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.087255    4515 generic.go:356] "Generic (PLEG): container finished" podID="f4f9e120-ed62-40f6-9407-101232be2e4e" containerID="b6de1f68a0f87b46d78a2497f48af719f8b27df1722b0c211982153d1d2f52c4" exitCode=0
Nov 02 07:35:12.087476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.087348    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" event={"ID":"f4f9e120-ed62-40f6-9407-101232be2e4e","Type":"ContainerDied","Data":"b6de1f68a0f87b46d78a2497f48af719f8b27df1722b0c211982153d1d2f52c4"}
Nov 02 07:35:12.088717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.088689    4515 generic.go:356] "Generic (PLEG): container finished" podID="e6840fad-7fd6-4581-9cbd-626954c81911" containerID="8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b" exitCode=0
Nov 02 07:35:12.088848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.088742    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"
Nov 02 07:35:12.088848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.088777    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" event={"ID":"e6840fad-7fd6-4581-9cbd-626954c81911","Type":"ContainerDied","Data":"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"}
Nov 02 07:35:12.088848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.088834    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4" event={"ID":"e6840fad-7fd6-4581-9cbd-626954c81911","Type":"ContainerDied","Data":"97723a43368e3638f4581908f7d5173e9cc4ec1bc8c2b31537647b615d840408"}
Nov 02 07:35:12.089033 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.088861    4515 scope.go:122] "RemoveContainer" containerID="8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"
Nov 02 07:35:12.098397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.098382    4515 scope.go:122] "RemoveContainer" containerID="8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"
Nov 02 07:35:12.098699 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:12.098670    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b\": container with ID starting with 8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b not found: ID does not exist" containerID="8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"
Nov 02 07:35:12.098809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.098707    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b"} err="failed to get container status \"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b\": rpc error: code = NotFound desc = could not find container \"8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b\": container with ID starting with 8d8861fdd54983882a1a6d918ad3bc412f540da900169f3acfebce4674aca88b not found: ID does not exist"
Nov 02 07:35:12.103143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.103125    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:35:12.187958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.187877    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:35:12.190810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.190779    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9780053-78d887b485-s52j4"]
Nov 02 07:35:12.235907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.235869    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\") pod \"f4f9e120-ed62-40f6-9407-101232be2e4e\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") "
Nov 02 07:35:12.236110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.235955    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\") pod \"f4f9e120-ed62-40f6-9407-101232be2e4e\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") "
Nov 02 07:35:12.236110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.236061    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\" (UniqueName: \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\") pod \"f4f9e120-ed62-40f6-9407-101232be2e4e\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") "
Nov 02 07:35:12.236232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.236130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\") pod \"f4f9e120-ed62-40f6-9407-101232be2e4e\" (UID: \"f4f9e120-ed62-40f6-9407-101232be2e4e\") "
Nov 02 07:35:12.236358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.236333    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca" pod "f4f9e120-ed62-40f6-9407-101232be2e4e" (UID: "f4f9e120-ed62-40f6-9407-101232be2e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:12.236516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.236503    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f9e120-ed62-40f6-9407-101232be2e4e-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.238184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.238159    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls" pod "f4f9e120-ed62-40f6-9407-101232be2e4e" (UID: "f4f9e120-ed62-40f6-9407-101232be2e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:12.238329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.238217    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs" pod "f4f9e120-ed62-40f6-9407-101232be2e4e" (UID: "f4f9e120-ed62-40f6-9407-101232be2e4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:12.238329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.238221    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx" pod "f4f9e120-ed62-40f6-9407-101232be2e4e" (UID: "f4f9e120-ed62-40f6-9407-101232be2e4e"). InnerVolumeSpecName "kube-api-access-zk7nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:12.337349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.337291    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.337349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.337344    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7nx\" (UniqueName: \"kubernetes.io/projected/f4f9e120-ed62-40f6-9407-101232be2e4e-kube-api-access-zk7nx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.337889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:12.337360    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f9e120-ed62-40f6-9407-101232be2e4e-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:13.096540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.096499    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w" event={"ID":"f4f9e120-ed62-40f6-9407-101232be2e4e","Type":"ContainerDied","Data":"075d78abf074697596294d81585bbae5260bdfe7a13135ca7625a5f73f36ac85"}
Nov 02 07:35:13.096755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.096509    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"
Nov 02 07:35:13.096755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.096603    4515 scope.go:122] "RemoveContainer" containerID="b6de1f68a0f87b46d78a2497f48af719f8b27df1722b0c211982153d1d2f52c4"
Nov 02 07:35:13.148703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.148671    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:35:13.161180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.161151    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ff8e809-58bb7648-jgh4w"]
Nov 02 07:35:13.374022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.373944    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6840fad-7fd6-4581-9cbd-626954c81911" path="/var/lib/kubelet/pods/e6840fad-7fd6-4581-9cbd-626954c81911/volumes"
Nov 02 07:35:13.374407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.374368    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f9e120-ed62-40f6-9407-101232be2e4e" path="/var/lib/kubelet/pods/f4f9e120-ed62-40f6-9407-101232be2e4e/volumes"
Nov 02 07:35:13.376835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.376816    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:35:13.377012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:13.376995    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" containerID="cri-o://14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca" gracePeriod=30
Nov 02 07:35:15.738441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:15.738361    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:16.044874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.044810    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" podUID="f9ceed76-f654-476c-99b4-36833ed6ecaa" containerName="payload-pre-processing" containerID="cri-o://03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9" gracePeriod=25
Nov 02 07:35:16.151787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.151747    4515 patch_prober.go:29] interesting pod/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:35:16.151951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.151809    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:35:16.304778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.304718    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:35:16.479600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.479565    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\") pod \"f9ceed76-f654-476c-99b4-36833ed6ecaa\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") "
Nov 02 07:35:16.479830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.479663    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\" (UniqueName: \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\") pod \"f9ceed76-f654-476c-99b4-36833ed6ecaa\" (UID: \"f9ceed76-f654-476c-99b4-36833ed6ecaa\") "
Nov 02 07:35:16.480006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.479982    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume" pod "f9ceed76-f654-476c-99b4-36833ed6ecaa" (UID: "f9ceed76-f654-476c-99b4-36833ed6ecaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:16.480079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.480062    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ceed76-f654-476c-99b4-36833ed6ecaa-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:16.481987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.481953    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8" pod "f9ceed76-f654-476c-99b4-36833ed6ecaa" (UID: "f9ceed76-f654-476c-99b4-36833ed6ecaa"). InnerVolumeSpecName "kube-api-access-fznj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:16.582098 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.582007    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fznj8\" (UniqueName: \"kubernetes.io/projected/f9ceed76-f654-476c-99b4-36833ed6ecaa-kube-api-access-fznj8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:16.667337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.667289    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:35:16.667546 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.667518    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" podUID="cc028b96-6567-4257-9a75-67b6a4e4313b" containerName="maas-api" containerID="cri-o://39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5" gracePeriod=30
Nov 02 07:35:16.903264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.903238    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:35:16.995448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.995414    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\" (UniqueName: \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\") pod \"cc028b96-6567-4257-9a75-67b6a4e4313b\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") "
Nov 02 07:35:16.995614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.995500    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\") pod \"cc028b96-6567-4257-9a75-67b6a4e4313b\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") "
Nov 02 07:35:16.995663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.995644    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\") pod \"cc028b96-6567-4257-9a75-67b6a4e4313b\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") "
Nov 02 07:35:16.995711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.995696    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\") pod \"cc028b96-6567-4257-9a75-67b6a4e4313b\" (UID: \"cc028b96-6567-4257-9a75-67b6a4e4313b\") "
Nov 02 07:35:16.996023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.995993    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca" pod "cc028b96-6567-4257-9a75-67b6a4e4313b" (UID: "cc028b96-6567-4257-9a75-67b6a4e4313b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:16.997788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.997760    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs" pod "cc028b96-6567-4257-9a75-67b6a4e4313b" (UID: "cc028b96-6567-4257-9a75-67b6a4e4313b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:16.998085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.998063    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls" pod "cc028b96-6567-4257-9a75-67b6a4e4313b" (UID: "cc028b96-6567-4257-9a75-67b6a4e4313b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:16.998149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:16.998118    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px" pod "cc028b96-6567-4257-9a75-67b6a4e4313b" (UID: "cc028b96-6567-4257-9a75-67b6a4e4313b"). InnerVolumeSpecName "kube-api-access-sc7px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:17.097390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.097120    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.097390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.097172    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7px\" (UniqueName: \"kubernetes.io/projected/cc028b96-6567-4257-9a75-67b6a4e4313b-kube-api-access-sc7px\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.097390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.097187    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc028b96-6567-4257-9a75-67b6a4e4313b-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.097390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.097199    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc028b96-6567-4257-9a75-67b6a4e4313b-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.143329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.139655    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:35:17.143329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.139950    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" containerID="cri-o://1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d" gracePeriod=30
Nov 02 07:35:17.145929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.145897    4515 generic.go:356] "Generic (PLEG): container finished" podID="cc028b96-6567-4257-9a75-67b6a4e4313b" containerID="39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5" exitCode=0
Nov 02 07:35:17.146166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.145942    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"
Nov 02 07:35:17.157945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.145979    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" event={"ID":"cc028b96-6567-4257-9a75-67b6a4e4313b","Type":"ContainerDied","Data":"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"}
Nov 02 07:35:17.158658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.158580    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz" event={"ID":"cc028b96-6567-4257-9a75-67b6a4e4313b","Type":"ContainerDied","Data":"06bd694f632d9d7b832f47cb79bff07b70e0b156fb676a7b49e35282a8d28530"}
Nov 02 07:35:17.158823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.158809    4515 scope.go:122] "RemoveContainer" containerID="39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"
Nov 02 07:35:17.160635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.160477    4515 generic.go:356] "Generic (PLEG): container finished" podID="f9ceed76-f654-476c-99b4-36833ed6ecaa" containerID="03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9" exitCode=0
Nov 02 07:35:17.160635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.160538    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"
Nov 02 07:35:17.160635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.160568    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" event={"ID":"f9ceed76-f654-476c-99b4-36833ed6ecaa","Type":"ContainerDied","Data":"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"}
Nov 02 07:35:17.160635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.160620    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz" event={"ID":"f9ceed76-f654-476c-99b4-36833ed6ecaa","Type":"ContainerDied","Data":"064cd6e3e2244a66fe89ba5cab8c9e2c5c4c4910776c29c553bd55b61bb61615"}
Nov 02 07:35:17.173000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.172974    4515 scope.go:122] "RemoveContainer" containerID="39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"
Nov 02 07:35:17.173338 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:17.173293    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5\": container with ID starting with 39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5 not found: ID does not exist" containerID="39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"
Nov 02 07:35:17.173423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.173338    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5"} err="failed to get container status \"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5\": rpc error: code = NotFound desc = could not find container \"39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5\": container with ID starting with 39c7ace8dc3de8defd6908b323413c018e6fafdba90b02f633333a6a51cd69d5 not found: ID does not exist"
Nov 02 07:35:17.173423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.173359    4515 scope.go:122] "RemoveContainer" containerID="03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"
Nov 02 07:35:17.181604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.181586    4515 scope.go:122] "RemoveContainer" containerID="03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"
Nov 02 07:35:17.181850 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:17.181834    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9\": container with ID starting with 03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9 not found: ID does not exist" containerID="03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"
Nov 02 07:35:17.181903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.181856    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9"} err="failed to get container status \"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9\": rpc error: code = NotFound desc = could not find container \"03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9\": container with ID starting with 03a00e33a1363aa15e82cc6f9d4cf861bb0233b89344dd1ea90fbc84b6b58fe9 not found: ID does not exist"
Nov 02 07:35:17.224988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.224863    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:35:17.230007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.229969    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-be1fb7c9-6bc8f849b7-fwxkz"]
Nov 02 07:35:17.267128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.267095    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:35:17.270897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.270867    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-935d4e21-65dbf594cf-z8qnz"]
Nov 02 07:35:17.372626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.372543    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc028b96-6567-4257-9a75-67b6a4e4313b" path="/var/lib/kubelet/pods/cc028b96-6567-4257-9a75-67b6a4e4313b/volumes"
Nov 02 07:35:17.373059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.373033    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ceed76-f654-476c-99b4-36833ed6ecaa" path="/var/lib/kubelet/pods/f9ceed76-f654-476c-99b4-36833ed6ecaa/volumes"
Nov 02 07:35:17.750252 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:17.750183    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:35:17.897620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:17.897581    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:35:18.333494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.333458    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:35:18.334071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334050    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e6840fad-7fd6-4581-9cbd-626954c81911" containerName="maas-api"
Nov 02 07:35:18.334116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334074    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6840fad-7fd6-4581-9cbd-626954c81911" containerName="maas-api"
Nov 02 07:35:18.334116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334100    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f4f9e120-ed62-40f6-9407-101232be2e4e" containerName="maas-api"
Nov 02 07:35:18.334116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334109    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f9e120-ed62-40f6-9407-101232be2e4e" containerName="maas-api"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334129    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f847af58-af1f-418f-8969-8cefd1889729" containerName="payload-pre-processing"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334137    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f847af58-af1f-418f-8969-8cefd1889729" containerName="payload-pre-processing"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334168    4515 cpu_manager.go:414] "containerMap: removing container" podUID="cc028b96-6567-4257-9a75-67b6a4e4313b" containerName="maas-api"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334175    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc028b96-6567-4257-9a75-67b6a4e4313b" containerName="maas-api"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334193    4515 cpu_manager.go:414] "containerMap: removing container" podUID="749f6c1a-2475-4606-a5f0-4b297606ef71" containerName="payload-processing"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334201    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749f6c1a-2475-4606-a5f0-4b297606ef71" containerName="payload-processing"
Nov 02 07:35:18.334215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334212    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f9ceed76-f654-476c-99b4-36833ed6ecaa" containerName="payload-pre-processing"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334220    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ceed76-f654-476c-99b4-36833ed6ecaa" containerName="payload-pre-processing"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334329    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc028b96-6567-4257-9a75-67b6a4e4313b" containerName="maas-api"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334344    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="749f6c1a-2475-4606-a5f0-4b297606ef71" containerName="payload-processing"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334355    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ceed76-f654-476c-99b4-36833ed6ecaa" containerName="payload-pre-processing"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334364    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6840fad-7fd6-4581-9cbd-626954c81911" containerName="maas-api"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334375    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f9e120-ed62-40f6-9407-101232be2e4e" containerName="maas-api"
Nov 02 07:35:18.334458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.334387    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f847af58-af1f-418f-8969-8cefd1889729" containerName="payload-pre-processing"
Nov 02 07:35:18.339210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.339190    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.358662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.358602    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-337764-openshift-default-dockercfg-45fm2\""
Nov 02 07:35:18.359902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.359863    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:35:18.429920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.429884    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.429926    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhdk\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430013    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430072    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430196    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430244    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430351    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430384    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.430555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.430539    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531563    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531595    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531637    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531679    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhdk\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531727    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531766    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531813    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.531847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.531831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.532336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.532012    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.532336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.532042    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.532436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.532303    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.533117 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.533093    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.533624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.533600    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.534282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.534259    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.534527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.534505    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.562713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.562678    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.562831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.562788    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhdk\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.645169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.645132    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:35:18.653932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.653908    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733199    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733236    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733263    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733288    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733321    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733352    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733415    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733431    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.733946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.733462    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\") pod \"40c67379-ce1d-4477-ab19-fb01f8359db0\" (UID: \"40c67379-ce1d-4477-ab19-fb01f8359db0\") "
Nov 02 07:35:18.735164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.734647    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:18.735164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.734674    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:18.735164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.734851    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:18.735164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.735094    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:18.735164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.735146    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:18.737210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.737095    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:35:18.737210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.737190    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:18.737345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.737209    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "kube-api-access-vvlj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:18.738410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.738390    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy" pod "40c67379-ce1d-4477-ab19-fb01f8359db0" (UID: "40c67379-ce1d-4477-ab19-fb01f8359db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:18.834467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834431    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834461    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlj9\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-kube-api-access-vvlj9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834472    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834481    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834489    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834497    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c67379-ce1d-4477-ab19-fb01f8359db0-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834504    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834511    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.834701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.834520    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c67379-ce1d-4477-ab19-fb01f8359db0-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:18.978771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.978744    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:35:18.981003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.980972    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:18.981102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.981036    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:18.981102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:18.981066    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:19.175480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.175440    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerStarted","Data":"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"}
Nov 02 07:35:19.175686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.175512    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerStarted","Data":"89b35631b6db791090997efdcdb2d04656880fce22ce81042c1f77cfa12079a7"}
Nov 02 07:35:19.177587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.177556    4515 generic.go:356] "Generic (PLEG): container finished" podID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerID="14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca" exitCode=0
Nov 02 07:35:19.177724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.177612    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"
Nov 02 07:35:19.177724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.177632    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerDied","Data":"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"}
Nov 02 07:35:19.177724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.177675    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574" event={"ID":"40c67379-ce1d-4477-ab19-fb01f8359db0","Type":"ContainerDied","Data":"6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4"}
Nov 02 07:35:19.177724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.177697    4515 scope.go:122] "RemoveContainer" containerID="14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"
Nov 02 07:35:19.186933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.186902    4515 scope.go:122] "RemoveContainer" containerID="2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"
Nov 02 07:35:19.197379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.197352    4515 scope.go:122] "RemoveContainer" containerID="14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"
Nov 02 07:35:19.197720 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:19.197699    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca\": container with ID starting with 14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca not found: ID does not exist" containerID="14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"
Nov 02 07:35:19.197825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.197731    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca"} err="failed to get container status \"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca\": rpc error: code = NotFound desc = could not find container \"14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca\": container with ID starting with 14a25ec3eee645c29b9fcfa5aebada1fd90e454ef9a9c7084a424036fc1222ca not found: ID does not exist"
Nov 02 07:35:19.197825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.197755    4515 scope.go:122] "RemoveContainer" containerID="2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"
Nov 02 07:35:19.198017 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:19.197998    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17\": container with ID starting with 2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17 not found: ID does not exist" containerID="2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"
Nov 02 07:35:19.198055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.198024    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17"} err="failed to get container status \"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17\": rpc error: code = NotFound desc = could not find container \"2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17\": container with ID starting with 2a2cee7065f4a5db8d94f784071b7b9c30a895f21e04fb45d8491ef66400bd17 not found: ID does not exist"
Nov 02 07:35:19.222065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.222016    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podStartSLOduration=1.222001037 podStartE2EDuration="1.222001037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:19.220837723 +0000 UTC m=+663.983274658" watchObservedRunningTime="2026-11-02 07:35:19.222001037 +0000 UTC m=+663.984437945"
Nov 02 07:35:19.258691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.258660    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:35:19.270880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.270850    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8ff8e809-openshift-default-6b877dd89c-ln574"]
Nov 02 07:35:19.372340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.372286    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" path="/var/lib/kubelet/pods/40c67379-ce1d-4477-ab19-fb01f8359db0/volumes"
Nov 02 07:35:19.655010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.654862    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:19.682850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.682822    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 07:35:19.683278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.683262    4515 cpu_manager.go:414] "containerMap: removing container" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.683387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.683280    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.683444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.683413    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.690430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.690406    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.706678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.706651    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 07:35:19.706896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.706880    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-354660\""
Nov 02 07:35:19.709607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.709563    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-354660-dockercfg-nnscf\""
Nov 02 07:35:19.742931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.742900    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gk9\" (UniqueName: \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.743086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.742944    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.778930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.778900    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 07:35:19.779298 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.779284    4515 cpu_manager.go:414] "containerMap: removing container" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.779351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.779299    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.779415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.779405    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c67379-ce1d-4477-ab19-fb01f8359db0" containerName="istio-proxy"
Nov 02 07:35:19.782496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.782478    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.844418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.844374    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gk9\" (UniqueName: \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.844616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.844435    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.844616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.844478    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j25fv\" (UniqueName: \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.844616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.844532    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.846601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.846059    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.848446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.848420    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 07:35:19.868935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.868898    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gk9\" (UniqueName: \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\") pod \"payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:19.945649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.945567    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j25fv\" (UniqueName: \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.945649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.945615    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.946272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.946254    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:19.972655 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:19.972618    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j25fv\" (UniqueName: \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\") pod \"payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:20.003850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.003817    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:20.107381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.107343    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:20.302070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.302038    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 07:35:20.408380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.408352    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 07:35:20.562167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.562086    4515 patch_prober.go:29] interesting pod/e2e-ait-935d4e21-openshift-default-555f96899-bfw85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:35:20.562167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.562143    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:35:20.659356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.659300    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:20.886471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.886398    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 07:35:20.890432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.890405    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.912148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.911933    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9db802-openshift-default-dockercfg-4p9sf\""
Nov 02 07:35:20.915200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.915167    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 07:35:20.954728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.954693    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.954932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.954753    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.954932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.954807    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.954932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.954921    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.955084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.954967    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.955084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.955033    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.955084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.955076    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.955229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.955111    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w2n\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:20.955229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:20.955174    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056018    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056094    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056121    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056155    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056198    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056238    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056274    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056342    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w2n\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056390    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056575    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056852    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.056938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056874    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.057015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.056989    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.057074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.057020    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.058465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.058444    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.058613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.058597    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.079883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.079853    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.080468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.080445    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w2n\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.191731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.191628    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" event={"ID":"b7735656-137d-412b-8e56-4efb71a4f697","Type":"ContainerStarted","Data":"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"}
Nov 02 07:35:21.191731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.191718    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:21.191981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.191748    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" event={"ID":"b7735656-137d-412b-8e56-4efb71a4f697","Type":"ContainerStarted","Data":"90fb538e53d31e67d1ae17f21cb87b3c3c59c6eaafa529745f8fe86ab398a025"}
Nov 02 07:35:21.193449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.193419    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" event={"ID":"bbf7fff7-9562-4547-b7b9-735f3c2503cd","Type":"ContainerStarted","Data":"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"}
Nov 02 07:35:21.193577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.193474    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" event={"ID":"bbf7fff7-9562-4547-b7b9-735f3c2503cd","Type":"ContainerStarted","Data":"176fd895985b4d0eaff4a64fad9278eecc23c420c45ba45f37db5ef6ea50737d"}
Nov 02 07:35:21.193626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.193602    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:21.193678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.193645    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:21.194385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.194366    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:21.203502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.203483    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:21.236790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.236742    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" podStartSLOduration=2.236728822 podStartE2EDuration="2.236728822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:21.2352082 +0000 UTC m=+665.997645108" watchObservedRunningTime="2026-11-02 07:35:21.236728822 +0000 UTC m=+665.999165727"
Nov 02 07:35:21.324063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.323997    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" podStartSLOduration=2.323978896 podStartE2EDuration="2.323978896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:21.322375596 +0000 UTC m=+666.084812530" watchObservedRunningTime="2026-11-02 07:35:21.323978896 +0000 UTC m=+666.086415801"
Nov 02 07:35:21.443423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.443083    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:21.533707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.524188    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 07:35:21.533707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.526156    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:21.533707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.526236    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:21.533707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:21.526276    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:22.200392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.200363    4515 generic.go:356] "Generic (PLEG): container finished" podID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerID="1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d" exitCode=0
Nov 02 07:35:22.200552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.200435    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerDied","Data":"1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d"}
Nov 02 07:35:22.200552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.200502    4515 scope.go:122] "RemoveContainer" containerID="584688a1e76f97355e1e1e4ef95c1ac556a11b641fb2d46c38082ba82d627549"
Nov 02 07:35:22.202086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.202062    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerStarted","Data":"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"}
Nov 02 07:35:22.202181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.202114    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerStarted","Data":"625beb8c87792e125f9c250ecf2668e5388caab2d98b3d1e547274310cad1260"}
Nov 02 07:35:22.203609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.203588    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:22.207961 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:22.207924    4515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da38b4_f74b_4794_b57e_38ea99cea1e0.slice/crio-1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c67379_ce1d_4477_ab19_fb01f8359db0.slice/crio-6652bbdc0de6183c892956bfd7bf114e9aa696f6ab3a3db8d6e988a4e1e338c4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da38b4_f74b_4794_b57e_38ea99cea1e0.slice/crio-conmon-1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 07:35:22.209765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.209744    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:22.260705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.260655    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podStartSLOduration=2.260640994 podStartE2EDuration="2.260640994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:22.260296182 +0000 UTC m=+667.022733089" watchObservedRunningTime="2026-11-02 07:35:22.260640994 +0000 UTC m=+667.023077899"
Nov 02 07:35:22.814182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.814155    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877023    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877101    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877156    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877193    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877247    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877350    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877414    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877447    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877505    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\") pod \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\" (UID: \"b5da38b4-f74b-4794-b57e-38ea99cea1e0\") "
Nov 02 07:35:22.877978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.877926    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:22.878120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.878100    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:22.878346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.878328    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:22.878475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.878460    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:22.878622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.878601    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:22.880964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.880933    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:22.881440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.881413    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "kube-api-access-q5652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:22.881548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.881466    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:22.881719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.881689    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo" pod "b5da38b4-f74b-4794-b57e-38ea99cea1e0" (UID: "b5da38b4-f74b-4794-b57e-38ea99cea1e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:35:22.898867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.898821    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" podUID="6087761a-ce01-47e1-9203-0d9848d28892" containerName="payload-processing" containerID="cri-o://a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6" gracePeriod=25
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979000    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979033    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979044    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979053    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979062    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5652\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-kube-api-access-q5652\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979071    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979080    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979087    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.979089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:22.979095    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5da38b4-f74b-4794-b57e-38ea99cea1e0-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.142474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.142443    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:35:23.180577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.180550    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\") pod \"6087761a-ce01-47e1-9203-0d9848d28892\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") "
Nov 02 07:35:23.180716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.180625    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\" (UniqueName: \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\") pod \"6087761a-ce01-47e1-9203-0d9848d28892\" (UID: \"6087761a-ce01-47e1-9203-0d9848d28892\") "
Nov 02 07:35:23.180967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.180933    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume" pod "6087761a-ce01-47e1-9203-0d9848d28892" (UID: "6087761a-ce01-47e1-9203-0d9848d28892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:23.181091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.181007    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6087761a-ce01-47e1-9203-0d9848d28892-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.183506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.183322    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg" pod "6087761a-ce01-47e1-9203-0d9848d28892" (UID: "6087761a-ce01-47e1-9203-0d9848d28892"). InnerVolumeSpecName "kube-api-access-2lgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:23.209014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.208979    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85" event={"ID":"b5da38b4-f74b-4794-b57e-38ea99cea1e0","Type":"ContainerDied","Data":"6339e76a712e745d033445c16c386e324357d47a8f8937c633d0bb7a180feb68"}
Nov 02 07:35:23.209173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.208992    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"
Nov 02 07:35:23.209173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.209075    4515 scope.go:122] "RemoveContainer" containerID="1768b90be07577fbe241adad7d4b57563fed0db9b9ec5b026acb42a0d9b9da6d"
Nov 02 07:35:23.210542 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.210515    4515 generic.go:356] "Generic (PLEG): container finished" podID="6087761a-ce01-47e1-9203-0d9848d28892" containerID="a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6" exitCode=0
Nov 02 07:35:23.210678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.210565    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"
Nov 02 07:35:23.210678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.210597    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" event={"ID":"6087761a-ce01-47e1-9203-0d9848d28892","Type":"ContainerDied","Data":"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"}
Nov 02 07:35:23.210678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.210653    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n" event={"ID":"6087761a-ce01-47e1-9203-0d9848d28892","Type":"ContainerDied","Data":"e1996cb4e709f719f35e47563cee315430c5dc2de22891d5df4318a17742ab66"}
Nov 02 07:35:23.211184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.210985    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:23.212048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.212028    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:23.219369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.219134    4515 scope.go:122] "RemoveContainer" containerID="a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"
Nov 02 07:35:23.227470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.227446    4515 scope.go:122] "RemoveContainer" containerID="a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"
Nov 02 07:35:23.227725 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:23.227705    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6\": container with ID starting with a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6 not found: ID does not exist" containerID="a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"
Nov 02 07:35:23.227767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.227736    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6"} err="failed to get container status \"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6\": rpc error: code = NotFound desc = could not find container \"a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6\": container with ID starting with a72730eee9c4782f963961618c9fe6213de83c6902d81f6423a99210776638f6 not found: ID does not exist"
Nov 02 07:35:23.262286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.262253    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:35:23.266196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.266147    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c43c08e-56884bd7b7-v9m2n"]
Nov 02 07:35:23.281923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.281807    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lgkg\" (UniqueName: \"kubernetes.io/projected/6087761a-ce01-47e1-9203-0d9848d28892-kube-api-access-2lgkg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.301691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.301660    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:35:23.313912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.313885    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-935d4e21-openshift-default-555f96899-bfw85"]
Nov 02 07:35:23.376167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.376134    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6087761a-ce01-47e1-9203-0d9848d28892" path="/var/lib/kubelet/pods/6087761a-ce01-47e1-9203-0d9848d28892/volumes"
Nov 02 07:35:23.376600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.376586    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" path="/var/lib/kubelet/pods/b5da38b4-f74b-4794-b57e-38ea99cea1e0/volumes"
Nov 02 07:35:23.419202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.419168    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:35:23.419429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.419406    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" podUID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" containerName="maas-api" containerID="cri-o://e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268" gracePeriod=30
Nov 02 07:35:23.691371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.691346    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:35:23.786786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.786705    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\") pod \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") "
Nov 02 07:35:23.786938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.786910    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") pod \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") "
Nov 02 07:35:23.787027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.787010    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\") pod \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") "
Nov 02 07:35:23.787132 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.787044    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\" (UniqueName: \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\") pod \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\" (UID: \"2cf22a34-6257-4d2d-862a-a194ce0b32f8\") "
Nov 02 07:35:23.787382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.787355    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca" pod "2cf22a34-6257-4d2d-862a-a194ce0b32f8" (UID: "2cf22a34-6257-4d2d-862a-a194ce0b32f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:23.788819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.788795    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs" pod "2cf22a34-6257-4d2d-862a-a194ce0b32f8" (UID: "2cf22a34-6257-4d2d-862a-a194ce0b32f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:23.789034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.789004    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls" pod "2cf22a34-6257-4d2d-862a-a194ce0b32f8" (UID: "2cf22a34-6257-4d2d-862a-a194ce0b32f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:23.789129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.789058    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j" pod "2cf22a34-6257-4d2d-862a-a194ce0b32f8" (UID: "2cf22a34-6257-4d2d-862a-a194ce0b32f8"). InnerVolumeSpecName "kube-api-access-87s6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:23.890662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.888326    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.890662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.890412    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf22a34-6257-4d2d-862a-a194ce0b32f8-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.890662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.890506    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf22a34-6257-4d2d-862a-a194ce0b32f8-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.890662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:23.890548    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87s6j\" (UniqueName: \"kubernetes.io/projected/2cf22a34-6257-4d2d-862a-a194ce0b32f8-kube-api-access-87s6j\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:35:24.218221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.218141    4515 generic.go:356] "Generic (PLEG): container finished" podID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" containerID="e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268" exitCode=0
Nov 02 07:35:24.218221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.218185    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"
Nov 02 07:35:24.218437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.218225    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" event={"ID":"2cf22a34-6257-4d2d-862a-a194ce0b32f8","Type":"ContainerDied","Data":"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"}
Nov 02 07:35:24.218437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.218277    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g" event={"ID":"2cf22a34-6257-4d2d-862a-a194ce0b32f8","Type":"ContainerDied","Data":"11af97383ea637a3d9e284a620b3318cb6e153b20b980f65b9b5cb1ed07deb76"}
Nov 02 07:35:24.218437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.218337    4515 scope.go:122] "RemoveContainer" containerID="e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"
Nov 02 07:35:24.229699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.229677    4515 scope.go:122] "RemoveContainer" containerID="e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"
Nov 02 07:35:24.230351 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:24.230323    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268\": container with ID starting with e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268 not found: ID does not exist" containerID="e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"
Nov 02 07:35:24.230501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.230362    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268"} err="failed to get container status \"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268\": rpc error: code = NotFound desc = could not find container \"e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268\": container with ID starting with e1eccd5f753afc0b0427b6983a90d744104b02e5d31a4e0a323b2d7cbebf3268 not found: ID does not exist"
Nov 02 07:35:24.247327 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247291    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 07:35:24.247687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247674    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247688    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247702    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" containerName="maas-api"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247708    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" containerName="maas-api"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247716    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247720    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247745    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6087761a-ce01-47e1-9203-0d9848d28892" containerName="payload-processing"
Nov 02 07:35:24.247768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247751    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6087761a-ce01-47e1-9203-0d9848d28892" containerName="payload-processing"
Nov 02 07:35:24.248087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247815    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6087761a-ce01-47e1-9203-0d9848d28892" containerName="payload-processing"
Nov 02 07:35:24.248087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247822    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.248087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247829    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" containerName="maas-api"
Nov 02 07:35:24.248087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.247834    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5da38b4-f74b-4794-b57e-38ea99cea1e0" containerName="istio-proxy"
Nov 02 07:35:24.251989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.251971    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.271197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.271161    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 07:35:24.273032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.273002    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-337764-dockercfg-hcdzp\""
Nov 02 07:35:24.274251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.274230    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-337764\""
Nov 02 07:35:24.280797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.280772    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:35:24.289558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.289534    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c43c08e-794cddbd7c-stm6g"]
Nov 02 07:35:24.294530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.294499    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbt7h\" (UniqueName: \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.294670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.294648    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.326005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.325976    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 07:35:24.329753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.329727    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.350773 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.350742    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 07:35:24.396333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.396274    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.396333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.396335    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.396543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.396364    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bxp\" (UniqueName: \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.396543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.396498    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbt7h\" (UniqueName: \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.397004 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.396985    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.426772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.426741    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbt7h\" (UniqueName: \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\") pod \"payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.497184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.497092    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.497184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.497133    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bxp\" (UniqueName: \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.497923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.497894    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.521417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.521385    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bxp\" (UniqueName: \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\") pod \"payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.566366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.566333    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:24.642757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.642727    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:24.870551 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:35:24.870522    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ecc2e1_0598_4102_84fc_ae6b6303660d.slice/crio-3c71cf2ed32b32a18d7a24457a09622f28516279a78ead31cbebec9956aa4b23 WatchSource:0}: Error finding container 3c71cf2ed32b32a18d7a24457a09622f28516279a78ead31cbebec9956aa4b23: Status 404 returned error can't find the container with id 3c71cf2ed32b32a18d7a24457a09622f28516279a78ead31cbebec9956aa4b23
Nov 02 07:35:24.871000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.870976    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 07:35:24.949943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:24.949900    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 07:35:24.950397 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:35:24.950368    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf990a90_7f57_4df0_967f_2f662229390c.slice/crio-c4b067a331adb3819043e0ad91ceaa3460949ab1dbbe5ca26c18ad83b32ab5b1 WatchSource:0}: Error finding container c4b067a331adb3819043e0ad91ceaa3460949ab1dbbe5ca26c18ad83b32ab5b1: Status 404 returned error can't find the container with id c4b067a331adb3819043e0ad91ceaa3460949ab1dbbe5ca26c18ad83b32ab5b1
Nov 02 07:35:25.227032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.226937    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" event={"ID":"60ecc2e1-0598-4102-84fc-ae6b6303660d","Type":"ContainerStarted","Data":"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"}
Nov 02 07:35:25.227032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.226996    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" event={"ID":"60ecc2e1-0598-4102-84fc-ae6b6303660d","Type":"ContainerStarted","Data":"3c71cf2ed32b32a18d7a24457a09622f28516279a78ead31cbebec9956aa4b23"}
Nov 02 07:35:25.227032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.227025    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:25.229634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.229604    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" event={"ID":"af990a90-7f57-4df0-967f-2f662229390c","Type":"ContainerStarted","Data":"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"}
Nov 02 07:35:25.229822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.229798    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:25.229938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.229847    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" event={"ID":"af990a90-7f57-4df0-967f-2f662229390c","Type":"ContainerStarted","Data":"c4b067a331adb3819043e0ad91ceaa3460949ab1dbbe5ca26c18ad83b32ab5b1"}
Nov 02 07:35:25.269619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.269561    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" podStartSLOduration=1.26954578 podStartE2EDuration="1.26954578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:25.268243853 +0000 UTC m=+670.030680759" watchObservedRunningTime="2026-11-02 07:35:25.26954578 +0000 UTC m=+670.031982686"
Nov 02 07:35:25.310534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.310469    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" podStartSLOduration=1.310453097 podStartE2EDuration="1.310453097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:25.309462035 +0000 UTC m=+670.071898942" watchObservedRunningTime="2026-11-02 07:35:25.310453097 +0000 UTC m=+670.072890002"
Nov 02 07:35:25.371878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:25.371845    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf22a34-6257-4d2d-862a-a194ce0b32f8" path="/var/lib/kubelet/pods/2cf22a34-6257-4d2d-862a-a194ce0b32f8/volumes"
Nov 02 07:35:26.153700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.153655    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:35:26.157844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.157823    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.174687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.174661    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:35:26.184565 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.184540    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7e7a83-openshift-default-dockercfg-8bmgw\""
Nov 02 07:35:26.214170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214141    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27654\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214177    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214195    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214249    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214378    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214433    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214467    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214515    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.214590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.214571    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315728    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315787    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315850    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27654\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315880    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315903    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.315952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315925    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.315953    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316053    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316147    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316299    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316677    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316720    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.316942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316919    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.317003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.316956    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.318432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.318408    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.318625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.318610    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.339985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.339943    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27654\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.339985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.339963    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.473734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.473647    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:26.789822 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:35:26.789791    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be33f92_ec9c_447a_ba57_7a2b31ffacc1.slice/crio-73b4976e7c04607eb370d1d7b86c2764fb111731ccebfefe296ed82e33d8884a WatchSource:0}: Error finding container 73b4976e7c04607eb370d1d7b86c2764fb111731ccebfefe296ed82e33d8884a: Status 404 returned error can't find the container with id 73b4976e7c04607eb370d1d7b86c2764fb111731ccebfefe296ed82e33d8884a
Nov 02 07:35:26.790108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.790084    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:35:26.792154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.792119    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:26.792239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.792186    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:26.792239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:26.792214    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:27.243793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:27.243708    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerStarted","Data":"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"}
Nov 02 07:35:27.243793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:27.243759    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerStarted","Data":"73b4976e7c04607eb370d1d7b86c2764fb111731ccebfefe296ed82e33d8884a"}
Nov 02 07:35:27.290691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:27.290633    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podStartSLOduration=1.2906190610000001 podStartE2EDuration="1.290619061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:27.289844358 +0000 UTC m=+672.052281287" watchObservedRunningTime="2026-11-02 07:35:27.290619061 +0000 UTC m=+672.053055966"
Nov 02 07:35:27.473779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:27.473748    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:27.479071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:27.479044    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:28.144443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.144362    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 07:35:28.147995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.147972    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.167101 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.167073    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-337764-serving-cert\""
Nov 02 07:35:28.179058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.179022    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 07:35:28.235812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.235771    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.235987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.235831    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtc86\" (UniqueName: \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.235987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.235932    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.236054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.235998    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.250689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.250664    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:28.251581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.251557    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:28.336999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.336951    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.337199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.337078    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.337199 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:28.337092    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-serving-cert: secret "maas-api-e2e-worker-w2-337764-serving-cert" not found
Nov 02 07:35:28.337199 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:28.337175    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls podName:71319f1c-381b-4f8d-94c4-cc44a2e3a7d9 nodeName:}" failed. No retries permitted until 2026-11-02 07:35:28.837146629 +0000 UTC m=+673.599583530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls") pod "maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" (UID: "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9") : secret "maas-api-e2e-worker-w2-337764-serving-cert" not found
Nov 02 07:35:28.337412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.337250    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.337412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.337352    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc86\" (UniqueName: \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.338507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.338479    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.343807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.343781    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.367996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.367952    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtc86\" (UniqueName: \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.841928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.841893    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:28.844382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:28.844355    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:29.068757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:29.068478    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:29.396406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:29.384482    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 07:35:29.411514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:35:29.401646    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71319f1c_381b_4f8d_94c4_cc44a2e3a7d9.slice/crio-76f6dcd160f9c408b711114ad2d0db0ed7ff7cda00b4e2c10a5684758e10d08f WatchSource:0}: Error finding container 76f6dcd160f9c408b711114ad2d0db0ed7ff7cda00b4e2c10a5684758e10d08f: Status 404 returned error can't find the container with id 76f6dcd160f9c408b711114ad2d0db0ed7ff7cda00b4e2c10a5684758e10d08f
Nov 02 07:35:30.262222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:30.262182    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" event={"ID":"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9","Type":"ContainerStarted","Data":"28e32109839975369eee286d45058deae4fbeca6b4021fc306cd13461d60a2d5"}
Nov 02 07:35:30.262427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:30.262293    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:30.262427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:30.262348    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" event={"ID":"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9","Type":"ContainerStarted","Data":"76f6dcd160f9c408b711114ad2d0db0ed7ff7cda00b4e2c10a5684758e10d08f"}
Nov 02 07:35:30.325880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:30.314432    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" podStartSLOduration=1.895369461 podStartE2EDuration="2.314411782s" totalImagesPullingTime="419.042321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:30.311268859 +0000 UTC m=+675.073705783" watchObservedRunningTime="2026-11-02 07:35:30.314411782 +0000 UTC m=+675.076848679"
Nov 02 07:35:31.746756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:31.746713    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:32.205582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:32.205548    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 07:35:32.205882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:32.205859    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 07:35:36.257765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:36.257729    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 07:35:36.260791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:36.260750    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 07:35:36.274109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:36.274084    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 07:35:37.346333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:37.340248    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:42.365029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.364939    4515 generic.go:356] "Generic (PLEG): container finished" podID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerID="cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed" exitCode=0
Nov 02 07:35:42.365029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.365008    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerDied","Data":"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"}
Nov 02 07:35:42.365769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.365484    4515 scope.go:122] "RemoveContainer" containerID="cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"
Nov 02 07:35:42.367037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.367009    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:42.367114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.367079    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:42.367157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:42.367117    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:43.380710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:43.380677    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerStarted","Data":"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"}
Nov 02 07:35:43.657820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:43.657724    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:43.658898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:43.658110    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 07:35:43.659143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:43.659114    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 07:35:44.654422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:44.654389    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 07:35:44.654906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:44.654451    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 07:35:45.654247 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:45.654210    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 07:35:45.654561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:45.654264    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 07:35:46.048583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:46.048545    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:46.654573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:46.654544    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 07:35:46.655061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:46.655022    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 07:35:47.655685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:47.655596    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 07:35:47.655685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:47.655660    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 07:35:47.762801 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:35:47.762725    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:35:48.408619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.408586    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerID="2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a" exitCode=0
Nov 02 07:35:48.408825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.408645    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerDied","Data":"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"}
Nov 02 07:35:48.409028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.409013    4515 scope.go:122] "RemoveContainer" containerID="2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"
Nov 02 07:35:48.410318 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.410258    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:48.410383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.410368    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:48.410422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.410407    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:48.773514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.773476    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:48.774435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.774410    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:48.776651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:48.776242    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:35:49.415785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:49.415747    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerStarted","Data":"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"}
Nov 02 07:35:50.204665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:50.204572    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:50.206111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:50.206087    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:35:50.206327 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:50.206287    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 07:35:51.204508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:51.204467    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:35:51.204691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:51.204531    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 07:35:52.203885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:52.203846    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:35:52.204148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:52.203898    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 07:35:53.204608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:53.204573    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:35:53.205003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:53.204631    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 07:35:53.211840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:53.211816    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:53.230886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:53.230850    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:35:54.204613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:54.204567    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:35:54.205009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:54.204642    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 07:35:55.209066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:55.209034    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:55.210060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:55.210042    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 07:35:57.512240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.512199    4515 generic.go:356] "Generic (PLEG): container finished" podID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerID="44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7" exitCode=0
Nov 02 07:35:57.512710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.512264    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerDied","Data":"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"}
Nov 02 07:35:57.512710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.512650    4515 scope.go:122] "RemoveContainer" containerID="44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"
Nov 02 07:35:57.514021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.513988    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:57.514086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.514073    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:57.514129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:57.514115    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:58.251417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:58.251376    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:58.543994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:58.543896    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerStarted","Data":"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"}
Nov 02 07:35:59.474412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:59.474374    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:35:59.475547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:59.475521    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 07:35:59.475652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:35:59.475569    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 07:36:00.474961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:00.474930    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 07:36:00.475377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:00.474981    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 07:36:01.475012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:01.474967    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 07:36:01.475568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:01.475054    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 07:36:02.474948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:02.474910    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 07:36:02.475226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:02.474973    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 07:36:03.474204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:03.474167    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 07:36:03.474406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:03.474239    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 07:36:04.478860 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.478827    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:36:04.479251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.479103    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:36:04.479754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.479736    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:36:04.706694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.706665    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 07:36:04.710590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.710566    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.730352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.730226    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"premium-w3-354660-kserve-self-signed-certs\""
Nov 02 07:36:04.730487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.730360    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"openshift-service-ca.crt\""
Nov 02 07:36:04.730487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.730452    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"kube-root-ca.crt\""
Nov 02 07:36:04.730487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.730454    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"default-dockercfg-kkdsp\""
Nov 02 07:36:04.732359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.732338    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 07:36:04.867538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867502    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.867707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867562    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc79q\" (UniqueName: \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.867707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867674    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.867779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867709    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.867779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867733    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.867891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.867869    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc79q\" (UniqueName: \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969387    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969409    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969427    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969625    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969676    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969799 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969780    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969850    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.969951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.969924    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.971781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.971761    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.971954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.971936    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:04.993436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:04.993357    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc79q\" (UniqueName: \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\") pod \"premium-w3-354660-kserve-79dc9d56cf-xwt78\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") " pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:05.039390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:05.039355    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:05.339453 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:36:05.339424    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243fd671_4009_4c9c_a604_3c557af7aa03.slice/crio-969008e92aba0dd5d06176cb9985227652a7e3a1eeef6b6777568ce1fd75bef5 WatchSource:0}: Error finding container 969008e92aba0dd5d06176cb9985227652a7e3a1eeef6b6777568ce1fd75bef5: Status 404 returned error can't find the container with id 969008e92aba0dd5d06176cb9985227652a7e3a1eeef6b6777568ce1fd75bef5
Nov 02 07:36:05.340177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:05.340151    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 07:36:05.588289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:05.588185    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" event={"ID":"243fd671-4009-4c9c-a604-3c557af7aa03","Type":"ContainerStarted","Data":"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"}
Nov 02 07:36:05.588289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:05.588235    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" event={"ID":"243fd671-4009-4c9c-a604-3c557af7aa03","Type":"ContainerStarted","Data":"969008e92aba0dd5d06176cb9985227652a7e3a1eeef6b6777568ce1fd75bef5"}
Nov 02 07:36:05.631873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:05.631823    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podStartSLOduration=1.631810707 podStartE2EDuration="1.631810707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:05.630399004 +0000 UTC m=+710.392835910" watchObservedRunningTime="2026-11-02 07:36:05.631810707 +0000 UTC m=+710.394247612"
Nov 02 07:36:06.040463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:06.040426    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:07.534115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:07.534081    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:36:09.715463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.715429    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 07:36:09.719849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.719821    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.738869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.738837    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 07:36:09.749741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.749714    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"default-dockercfg-z4w4q\""
Nov 02 07:36:09.749741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.749720    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"simulator-w1-7e7a83-kserve-self-signed-certs\""
Nov 02 07:36:09.812288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.812257    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"kube-root-ca.crt\""
Nov 02 07:36:09.812469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.812292    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"openshift-service-ca.crt\""
Nov 02 07:36:09.819827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.819800    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.819949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.819841    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4qj\" (UniqueName: \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.819949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.819864    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.820025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.819999    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.820062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.820028    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.820109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.820096    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.920840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.920802    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.920840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.920846    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.920866    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.920980    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.921023    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4qj\" (UniqueName: \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.921049    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.921200    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.921338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.921433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.921403    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.923103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.923084    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.923434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.923417    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:09.969796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:09.966478    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4qj\" (UniqueName: \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\") pod \"simulator-w1-7e7a83-kserve-966bfbb4c-bdst4\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") " pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:10.032101 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:10.032067    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:10.332789 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:36:10.332759    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f55e14_e075_4ea4_91f4_e53800132d1b.slice/crio-52da50223492e5a832e2bbcbf3b7db71eeb74d62f67db447170cf84553896c99 WatchSource:0}: Error finding container 52da50223492e5a832e2bbcbf3b7db71eeb74d62f67db447170cf84553896c99: Status 404 returned error can't find the container with id 52da50223492e5a832e2bbcbf3b7db71eeb74d62f67db447170cf84553896c99
Nov 02 07:36:10.332984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:10.332880    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 07:36:10.614341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:10.614217    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" event={"ID":"51f55e14-e075-4ea4-91f4-e53800132d1b","Type":"ContainerStarted","Data":"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"}
Nov 02 07:36:10.614341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:10.614260    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" event={"ID":"51f55e14-e075-4ea4-91f4-e53800132d1b","Type":"ContainerStarted","Data":"52da50223492e5a832e2bbcbf3b7db71eeb74d62f67db447170cf84553896c99"}
Nov 02 07:36:10.657584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:10.657525    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podStartSLOduration=1.657510851 podStartE2EDuration="1.657510851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:10.655708972 +0000 UTC m=+715.418145879" watchObservedRunningTime="2026-11-02 07:36:10.657510851 +0000 UTC m=+715.419947749"
Nov 02 07:36:11.032804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:11.032766    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:15.039917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:15.039863    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:15.053208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:15.053178    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:15.651910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:15.651876    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:36:17.723483 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:36:17.723446    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:36:20.033246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:20.033208    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:20.046062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:20.046033    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:20.676887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:20.676855    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:36:20.719912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:20.719877    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:36:31.576971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:31.576933    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:36:42.433572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.433527    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 07:36:42.438735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.438709    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.456537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.456505    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 07:36:42.466878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.466848    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"premium-w1-7e7a83-kserve-self-signed-certs\""
Nov 02 07:36:42.580349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580288    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.580533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580456    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.580533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580488    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.580533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580509    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.580715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580634    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpzb\" (UniqueName: \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.580759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.580732    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.681981    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682064    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682163    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682187    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682212    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682256    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpzb\" (UniqueName: \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682369    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682459    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.682590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.682547    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.684488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.684433    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.684759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.684738    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.705731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.705708    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpzb\" (UniqueName: \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\") pod \"premium-w1-7e7a83-kserve-95764d486-bbtjh\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") " pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:42.754485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:42.754447    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:43.054700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:43.054671    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 07:36:43.054700 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:36:43.054682    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc07ee5_9e04_4cbc_8d34_e4a898d4ba56.slice/crio-8b91ad27475c0fca18fc9e6cf77ebab7a13c322fd0239676806147c1aa8f38f5 WatchSource:0}: Error finding container 8b91ad27475c0fca18fc9e6cf77ebab7a13c322fd0239676806147c1aa8f38f5: Status 404 returned error can't find the container with id 8b91ad27475c0fca18fc9e6cf77ebab7a13c322fd0239676806147c1aa8f38f5
Nov 02 07:36:43.795203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:43.795165    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" event={"ID":"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56","Type":"ContainerStarted","Data":"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"}
Nov 02 07:36:43.795203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:43.795210    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" event={"ID":"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56","Type":"ContainerStarted","Data":"8b91ad27475c0fca18fc9e6cf77ebab7a13c322fd0239676806147c1aa8f38f5"}
Nov 02 07:36:43.837926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:43.837870    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podStartSLOduration=1.8378549720000001 podStartE2EDuration="1.837854972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:43.835946752 +0000 UTC m=+748.598383659" watchObservedRunningTime="2026-11-02 07:36:43.837854972 +0000 UTC m=+748.600291877"
Nov 02 07:36:44.754921 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:44.754883    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:46.050066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:46.050031    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:36:47.721755 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:36:47.721719    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:36:52.755577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:52.755543    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:52.768159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:52.768131    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:52.857954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:52.857920    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:36:58.875278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:58.875197    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 07:36:58.879861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:58.879833    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:58.899635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:58.899608    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-354660-openshift-default-dockercfg-kgbn5\""
Nov 02 07:36:58.911076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:58.911049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 07:36:59.065845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.065810    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd8d\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.065858    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.065880    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066015    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066063    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066093    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066227 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066120    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066227 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066189    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.066302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.066264    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167444    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd8d\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167494    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167561    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167604    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167636    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167821    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167867    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167906    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167911    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.167968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.167955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.168085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.168062    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.168239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.168147    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.168385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.168360    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.168491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.168386    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.170076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.170057    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.170243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.170223    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.191526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.191496    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.191626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.191598    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd8d\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.193331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.193299    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:36:59.535509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.535469    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 07:36:59.535619 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:36:59.535473    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d605b9_1283_437c_b85e_e7bc74264388.slice/crio-1a2842cf0c3d4ab0ad3a935483b126d2d35d1641b6ab6469e83168b72d906fb8 WatchSource:0}: Error finding container 1a2842cf0c3d4ab0ad3a935483b126d2d35d1641b6ab6469e83168b72d906fb8: Status 404 returned error can't find the container with id 1a2842cf0c3d4ab0ad3a935483b126d2d35d1641b6ab6469e83168b72d906fb8
Nov 02 07:36:59.537800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.537767    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:36:59.537896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.537831    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:36:59.537896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.537861    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:36:59.886447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.886348    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" event={"ID":"98d605b9-1283-437c-b85e-e7bc74264388","Type":"ContainerStarted","Data":"badf15720331329e6ce7332c3729435bac87ac53dbd46e48f829fe6fdd9d386e"}
Nov 02 07:36:59.886447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.886417    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" event={"ID":"98d605b9-1283-437c-b85e-e7bc74264388","Type":"ContainerStarted","Data":"1a2842cf0c3d4ab0ad3a935483b126d2d35d1641b6ab6469e83168b72d906fb8"}
Nov 02 07:36:59.962338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:36:59.962270    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podStartSLOduration=1.9622552149999999 podStartE2EDuration="1.962255215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:59.947280336 +0000 UTC m=+764.709717242" watchObservedRunningTime="2026-11-02 07:36:59.962255215 +0000 UTC m=+764.724692121"
Nov 02 07:37:00.194452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:00.194373    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:37:00.196161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:00.196138    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:00.196241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:00.196205    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:01.194252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:01.194221    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:01.194682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:01.194273    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:02.194186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:02.194150    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:02.194404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:02.194220    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:03.194137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:03.194099    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:03.194347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:03.194159    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:04.194630 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:04.194584    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:04.195040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:04.194652    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:05.193873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:05.193840    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 02 07:37:05.194084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:05.193898    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 02 07:37:06.240093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:06.240045    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:37:06.240780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:06.240754    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:37:06.241824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:06.241798    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 07:37:09.625825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:09.625787    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:37:17.722797 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:37:17.722756    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:37:21.660286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.660189    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"]
Nov 02 07:37:21.665055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.665037    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.682934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.682906    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"]
Nov 02 07:37:21.692494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.692459    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:37:21.692638 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.692543    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:37:21.733773 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.733733    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.733916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.733785    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvh7\" (UniqueName: \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.733975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.733935    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.835613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.835577    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.835775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.835738    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.835825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.835792    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvh7\" (UniqueName: \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.836488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.836461    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.838161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.838139    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.859480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.859442    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvh7\" (UniqueName: \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\") pod \"maas-api-revoke-sub-894bb881-f5lfb\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:21.979351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:21.979237    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:22.278803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:22.278775    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"]
Nov 02 07:37:22.280018 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:37:22.279991    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0799b7b9_5e28_45f2_ac0d_e6b3bda50762.slice/crio-706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c WatchSource:0}: Error finding container 706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c: Status 404 returned error can't find the container with id 706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c
Nov 02 07:37:23.011341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:23.011284    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb" event={"ID":"0799b7b9-5e28-45f2-ac0d-e6b3bda50762","Type":"ContainerStarted","Data":"706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c"}
Nov 02 07:37:26.030136 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:26.030099    4515 generic.go:356] "Generic (PLEG): container finished" podID="0799b7b9-5e28-45f2-ac0d-e6b3bda50762" containerID="4cca9a0d9f3caaa18deffe4b331d0a288bcd91cadc01f89cb3f9696bfd5b20d7" exitCode=0
Nov 02 07:37:26.030625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:26.030189    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb" event={"ID":"0799b7b9-5e28-45f2-ac0d-e6b3bda50762","Type":"ContainerDied","Data":"4cca9a0d9f3caaa18deffe4b331d0a288bcd91cadc01f89cb3f9696bfd5b20d7"}
Nov 02 07:37:27.166478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.166455    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:27.297358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.297249    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\") pod \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") "
Nov 02 07:37:27.297358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.297357    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\") pod \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") "
Nov 02 07:37:27.297608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.297410    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\" (UniqueName: \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\") pod \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\" (UID: \"0799b7b9-5e28-45f2-ac0d-e6b3bda50762\") "
Nov 02 07:37:27.297789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.297762    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca" pod "0799b7b9-5e28-45f2-ac0d-e6b3bda50762" (UID: "0799b7b9-5e28-45f2-ac0d-e6b3bda50762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:27.299508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.299486    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca" pod "0799b7b9-5e28-45f2-ac0d-e6b3bda50762" (UID: "0799b7b9-5e28-45f2-ac0d-e6b3bda50762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:27.299584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.299503    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7" pod "0799b7b9-5e28-45f2-ac0d-e6b3bda50762" (UID: "0799b7b9-5e28-45f2-ac0d-e6b3bda50762"). InnerVolumeSpecName "kube-api-access-5dvh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:27.398610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.398580    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-openshift-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:37:27.398610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.398610    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvh7\" (UniqueName: \"kubernetes.io/projected/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-kube-api-access-5dvh7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:37:27.398792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:27.398623    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0799b7b9-5e28-45f2-ac0d-e6b3bda50762-maas-api-xks-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:37:28.041934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:28.041893    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb" event={"ID":"0799b7b9-5e28-45f2-ac0d-e6b3bda50762","Type":"ContainerDied","Data":"706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c"}
Nov 02 07:37:28.041934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:28.041929    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"
Nov 02 07:37:28.042146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:28.041957    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706c388a83a67953de90d7bc78e60bfb3fc69fd2b07d15a516dd90781b5f8e7c"
Nov 02 07:37:34.253453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.253421    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:37:34.253887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.253874    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0799b7b9-5e28-45f2-ac0d-e6b3bda50762" containerName="revoke-keys"
Nov 02 07:37:34.253887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.253888    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0799b7b9-5e28-45f2-ac0d-e6b3bda50762" containerName="revoke-keys"
Nov 02 07:37:34.253962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.253956    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0799b7b9-5e28-45f2-ac0d-e6b3bda50762" containerName="revoke-keys"
Nov 02 07:37:34.257270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.257249    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.275132 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.275108    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0a310b1a5611\""
Nov 02 07:37:34.287893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.287860    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 07:37:34.287893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.287882    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 07:37:34.287893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.287894    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 07:37:34.288203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.287862    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 07:37:34.288203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.287859    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3845cbbf4190\""
Nov 02 07:37:34.288456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.288434    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ctb6j\""
Nov 02 07:37:34.290052 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.290033    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:37:34.305476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.305434    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.305645 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.305554    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbppc\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.305645 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.305627    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.305748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.305726    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.406543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.406507    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.407269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.407243    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.408232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.408212    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.408465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.408446    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbppc\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.408903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.408129    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.409799 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.409778    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.424328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.421058    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.433073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.433042    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbppc\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\") pod \"payload-processing-external-model-665db6c6c-slv9b\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") " pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.569879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.569779    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:34.882053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:34.882017    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:37:34.882983 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:37:34.882941    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf0a98_9d5b_40e3_86da_56bace0575b5.slice/crio-9b4c6795f55fada4702aa567a344da0cfb5bb770d12a6323529095fd4a3849a5 WatchSource:0}: Error finding container 9b4c6795f55fada4702aa567a344da0cfb5bb770d12a6323529095fd4a3849a5: Status 404 returned error can't find the container with id 9b4c6795f55fada4702aa567a344da0cfb5bb770d12a6323529095fd4a3849a5
Nov 02 07:37:35.080627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:35.080586    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" event={"ID":"0ddf0a98-9d5b-40e3-86da-56bace0575b5","Type":"ContainerStarted","Data":"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"}
Nov 02 07:37:35.080804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:35.080657    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" event={"ID":"0ddf0a98-9d5b-40e3-86da-56bace0575b5","Type":"ContainerStarted","Data":"9b4c6795f55fada4702aa567a344da0cfb5bb770d12a6323529095fd4a3849a5"}
Nov 02 07:37:35.080804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:35.080726    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:35.122697 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:35.122646    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" podStartSLOduration=1.122632864 podStartE2EDuration="1.122632864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:35.121489509 +0000 UTC m=+799.883926417" watchObservedRunningTime="2026-11-02 07:37:35.122632864 +0000 UTC m=+799.885069769"
Nov 02 07:37:41.063837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:41.063800    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:37:46.090382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:46.090353    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:37:47.723446 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:37:47.723405    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:37:48.132619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.132581    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:37:48.137782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.137761    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157141    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157220    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157289    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157382    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157445    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157536    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157599    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.157683 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.157655    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.168097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.168064    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:37:48.259171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259130    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259169    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259271    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259292    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259339    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259424    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259458    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259517    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259740    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259820    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.259944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259930    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.260019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.259999    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.260206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.260183    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.261734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.261712    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.262039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.262023    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.283549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.283520    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.283790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.283767    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-99lqk\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.452399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.452290    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:48.771082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.771050    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:37:48.771513 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:37:48.771492    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429a7df_22b6_456a_8ab7_383a0ac06aa4.slice/crio-b06c4fd3278a9da989bff0224ee162fefa9ccca508eba25de0f49e96aabf6c7c WatchSource:0}: Error finding container b06c4fd3278a9da989bff0224ee162fefa9ccca508eba25de0f49e96aabf6c7c: Status 404 returned error can't find the container with id b06c4fd3278a9da989bff0224ee162fefa9ccca508eba25de0f49e96aabf6c7c
Nov 02 07:37:48.773974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.773938    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:48.774096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.774021    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:48.774096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:48.774069    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:49.180851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.180746    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" event={"ID":"5429a7df-22b6-456a-8ab7-383a0ac06aa4","Type":"ContainerStarted","Data":"699e89403593006c4e0dade9afd6c4dab494d75ad9b2fe5c66846eb40719705a"}
Nov 02 07:37:49.180851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.180800    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" event={"ID":"5429a7df-22b6-456a-8ab7-383a0ac06aa4","Type":"ContainerStarted","Data":"b06c4fd3278a9da989bff0224ee162fefa9ccca508eba25de0f49e96aabf6c7c"}
Nov 02 07:37:49.241194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.241131    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podStartSLOduration=1.241115663 podStartE2EDuration="1.241115663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:49.23923073 +0000 UTC m=+814.001667637" watchObservedRunningTime="2026-11-02 07:37:49.241115663 +0000 UTC m=+814.003552653"
Nov 02 07:37:49.452904 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.452812    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:49.454117 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.454093    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:49.454184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:49.454153    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:50.453017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:50.452978    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:50.453465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:50.453040    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:51.453428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:51.453386    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:51.453956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:51.453441    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:52.458750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:52.458709    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:52.459465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:52.459434    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:53.454334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:53.454282    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:53.454519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:53.454369    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:54.453073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:54.453022    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:54.453510 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:54.453138    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:55.453628 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.453594    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:55.454037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.453664    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:55.808743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.808708    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:37:55.812445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.812422    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.831119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.831088    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.831276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.831182    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.831276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.831229    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x88z\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.831276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.831267    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.832676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.832654    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:37:55.932476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.932439    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.932737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.932541    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.932737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.932586    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x88z\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.932737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.932616    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.933432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.933410    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.933548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.933414    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.935150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.935120    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:55.958516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:55.958489    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x88z\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\") pod \"payload-processing-external-model-7898f6c747-89x6j\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") " pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:56.127120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:56.127028    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:56.438022 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:37:56.437984    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d2be5_7a51_4cf5_816c_cd30f91d98ec.slice/crio-328591a8975be9704084647819c20d43a265a94e8d4a8a2ee03c9515ec2596b7 WatchSource:0}: Error finding container 328591a8975be9704084647819c20d43a265a94e8d4a8a2ee03c9515ec2596b7: Status 404 returned error can't find the container with id 328591a8975be9704084647819c20d43a265a94e8d4a8a2ee03c9515ec2596b7
Nov 02 07:37:56.438601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:56.438530    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:37:56.453512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:56.453488    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 07:37:56.453616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:56.453532    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 07:37:57.244907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.244861    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" event={"ID":"e48d2be5-7a51-4cf5-816c-cd30f91d98ec","Type":"ContainerStarted","Data":"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"}
Nov 02 07:37:57.245391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.244942    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:37:57.245391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.244969    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" event={"ID":"e48d2be5-7a51-4cf5-816c-cd30f91d98ec","Type":"ContainerStarted","Data":"328591a8975be9704084647819c20d43a265a94e8d4a8a2ee03c9515ec2596b7"}
Nov 02 07:37:57.287996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.287930    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podStartSLOduration=2.287910131 podStartE2EDuration="2.287910131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:57.286516168 +0000 UTC m=+822.048953106" watchObservedRunningTime="2026-11-02 07:37:57.287910131 +0000 UTC m=+822.050347038"
Nov 02 07:37:57.438179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.438141    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:37:57.679619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.679582    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:57.679902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.679881    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:37:57.680749 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:37:57.680731    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:38:00.060147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.060111    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:38:00.070371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.070342    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.093411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.093370    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:38:00.104544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.104509    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0fbc17-openshift-default-dockercfg-p76f6\""
Nov 02 07:38:00.190395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190323    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190411    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190498    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190520    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190543    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190562    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmc9t\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190724    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190796    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.190900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.190846    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292455    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292516    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292544    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292575    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292638    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292665    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292698    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292725    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmc9t\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.292838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292766    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.293048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.292994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.293106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.293051    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.293169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.293109    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.293169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.293150    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.293548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.293528    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.295029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.295005    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.295226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.295209    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.316737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.316663    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.316865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.316839    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmc9t\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.386213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.386174    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:00.702530 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:38:00.702495    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7080f30d_98e2_4748_9c7a_c8ba02dd715c.slice/crio-75cddb65d1ec01748390cf1b724000d7e63fab19e896e277beb0a8569011bc16 WatchSource:0}: Error finding container 75cddb65d1ec01748390cf1b724000d7e63fab19e896e277beb0a8569011bc16: Status 404 returned error can't find the container with id 75cddb65d1ec01748390cf1b724000d7e63fab19e896e277beb0a8569011bc16
Nov 02 07:38:00.702827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.702804    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:38:00.705024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.704991    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:00.705115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.705057    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:00.705115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:00.705085    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:01.279882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.279846    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" event={"ID":"7080f30d-98e2-4748-9c7a-c8ba02dd715c","Type":"ContainerStarted","Data":"bbe778ed9004f986fa297378f12c702f362da5d512a50447c0ddde3b8c6213dd"}
Nov 02 07:38:01.280265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.279930    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" event={"ID":"7080f30d-98e2-4748-9c7a-c8ba02dd715c","Type":"ContainerStarted","Data":"75cddb65d1ec01748390cf1b724000d7e63fab19e896e277beb0a8569011bc16"}
Nov 02 07:38:01.334351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.334265    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podStartSLOduration=1.3342474979999999 podStartE2EDuration="1.334247498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:01.333825787 +0000 UTC m=+826.096262729" watchObservedRunningTime="2026-11-02 07:38:01.334247498 +0000 UTC m=+826.096684404"
Nov 02 07:38:01.386395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.386363    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:01.388122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.388096    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:01.388253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:01.388145    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:02.386789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:02.386742    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:02.387288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:02.386822    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:03.387384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:03.387349    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:03.387790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:03.387399    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:04.387240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:04.387205    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:04.387469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:04.387257    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:05.404331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:05.403464    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:05.404331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:05.403543    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:06.387123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:06.387091    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:06.387401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:06.387377    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:07.389818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:07.389782    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:38:07.390435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:07.390403    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 07:38:08.291961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:08.291918    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:08.874340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:08.874280    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:08.875634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:08.875604    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:08.876653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:08.876631    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:38:16.834535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:16.834498    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:38:17.723923 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:38:17.723878    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:38:18.254093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:18.254045    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:28.253872 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:28.253788    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:31.907129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:31.907094    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:38:34.265401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.265355    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 07:38:34.270333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.270287    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.286838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.286805    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 07:38:34.291997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.291966    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.292156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.292009    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.292156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.292076    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.292286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.292203    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.292286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.292235    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mz69\" (UniqueName: \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.292286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.292281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.297062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.297033    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"distinct-2-w1-7e7a83-kserve-self-signed-certs\""
Nov 02 07:38:34.393612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393574    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.393612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393617    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.393869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393758    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.393869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393796    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mz69\" (UniqueName: \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.393869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393855    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.394031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.393909    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.394094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.394049    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.394217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.394191    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.394355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.394320    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.396031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.396006    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.396261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.396242    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.417854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.417823    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mz69\" (UniqueName: \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\") pod \"distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.584036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.583953    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:34.900472 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:38:34.900441    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6de626b4_2ae0_4ebc_95d8_9eb7ea71605c.slice/crio-f5376b6af1a50450c5a79aa4ca1ac5ece63211e9081a15a694f24da995796a8d WatchSource:0}: Error finding container f5376b6af1a50450c5a79aa4ca1ac5ece63211e9081a15a694f24da995796a8d: Status 404 returned error can't find the container with id f5376b6af1a50450c5a79aa4ca1ac5ece63211e9081a15a694f24da995796a8d
Nov 02 07:38:34.900845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:34.900820    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 07:38:35.538446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:35.538409    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" event={"ID":"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c","Type":"ContainerStarted","Data":"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"}
Nov 02 07:38:35.538446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:35.538454    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" event={"ID":"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c","Type":"ContainerStarted","Data":"f5376b6af1a50450c5a79aa4ca1ac5ece63211e9081a15a694f24da995796a8d"}
Nov 02 07:38:35.583489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:35.583435    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podStartSLOduration=1.583419688 podStartE2EDuration="1.583419688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:35.581391458 +0000 UTC m=+860.343828365" watchObservedRunningTime="2026-11-02 07:38:35.583419688 +0000 UTC m=+860.345856594"
Nov 02 07:38:35.584526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:35.584503    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:38.253551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:38.253518    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:44.584526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:44.584491    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:44.597963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:44.597937    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:44.627105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:44.627071    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:38:45.923948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.923910    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 07:38:45.928825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.928796    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:45.945734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.945707    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"kube-root-ca.crt\""
Nov 02 07:38:45.958341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.958292    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"openshift-service-ca.crt\""
Nov 02 07:38:45.958747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.958701    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"distinct-2-w0-0fbc17-kserve-self-signed-certs\""
Nov 02 07:38:45.958964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.958943    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"default-dockercfg-k8rx2\""
Nov 02 07:38:45.962628 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:45.962608    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 07:38:46.028131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028100    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.028300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.028300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028273    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.028422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028409    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.028470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028446    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.028517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.028477    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfz56\" (UniqueName: \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142124    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142186    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142220    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfz56\" (UniqueName: \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142344    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142375    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.142607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142424    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.143321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.142844    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.143321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.143255    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.143675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.143653    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.150461 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.150420    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.150791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.150771    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.174081 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.173969    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfz56\" (UniqueName: \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\") pod \"distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.241378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.241350    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:46.549927 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:38:46.549899    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod307abf08_13a0_410e_bd8e_39a7c69d707f.slice/crio-427ede97cb06fbe40a3834028aa8e47257a5e0c37d8d427a1723805b881d59b0 WatchSource:0}: Error finding container 427ede97cb06fbe40a3834028aa8e47257a5e0c37d8d427a1723805b881d59b0: Status 404 returned error can't find the container with id 427ede97cb06fbe40a3834028aa8e47257a5e0c37d8d427a1723805b881d59b0
Nov 02 07:38:46.550336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.550298    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 07:38:46.629810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.629775    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" event={"ID":"307abf08-13a0-410e-bd8e-39a7c69d707f","Type":"ContainerStarted","Data":"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"}
Nov 02 07:38:46.629948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.629823    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" event={"ID":"307abf08-13a0-410e-bd8e-39a7c69d707f","Type":"ContainerStarted","Data":"427ede97cb06fbe40a3834028aa8e47257a5e0c37d8d427a1723805b881d59b0"}
Nov 02 07:38:46.673537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:46.673480    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podStartSLOduration=1.673466172 podStartE2EDuration="1.673466172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:46.671770726 +0000 UTC m=+871.434207633" watchObservedRunningTime="2026-11-02 07:38:46.673466172 +0000 UTC m=+871.435903078"
Nov 02 07:38:47.249333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:47.248457    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:47.723199 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:38:47.723114    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:38:48.125744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:48.125708    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:38:48.253469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:48.253434    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:52.923754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:52.923719    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:38:52.925041 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:52.925011    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:38:52.965949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:52.965913    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:38:52.970429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:52.970403    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:52.998574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:52.998543    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:38:53.041778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.041739    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.042014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.041793    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblrt\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.042014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.041934    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.042139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.042109    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.143442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.143405    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.143624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.143456    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.143624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.143490    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblrt\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.143624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.143531    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.144122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.144099    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.144269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.144248    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.145995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.145973    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.167421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.167393    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblrt\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\") pod \"payload-processing-external-model-66bbdd4b74-qhldk\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.284714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.284682    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.584874 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:38:53.584832    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eea6f5_aff5_4345_babb_61dcf06993e1.slice/crio-53f1aeff441cb33c18940c1f4c77c61c52328dd1a3968c84b8b09baf9f940fda WatchSource:0}: Error finding container 53f1aeff441cb33c18940c1f4c77c61c52328dd1a3968c84b8b09baf9f940fda: Status 404 returned error can't find the container with id 53f1aeff441cb33c18940c1f4c77c61c52328dd1a3968c84b8b09baf9f940fda
Nov 02 07:38:53.584997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.584952    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:38:53.668071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.668032    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" event={"ID":"e2eea6f5-aff5-4345-babb-61dcf06993e1","Type":"ContainerStarted","Data":"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"}
Nov 02 07:38:53.668250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.668093    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" event={"ID":"e2eea6f5-aff5-4345-babb-61dcf06993e1","Type":"ContainerStarted","Data":"53f1aeff441cb33c18940c1f4c77c61c52328dd1a3968c84b8b09baf9f940fda"}
Nov 02 07:38:53.668250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.668170    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:38:53.713940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:53.713883    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" podStartSLOduration=1.713870016 podStartE2EDuration="1.713870016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:53.711385357 +0000 UTC m=+878.473822263" watchObservedRunningTime="2026-11-02 07:38:53.713870016 +0000 UTC m=+878.476306922"
Nov 02 07:38:56.242062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:56.242027    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:56.254747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:56.254723    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:56.695170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:56.695141    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:38:57.924217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:57.924179    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing" containerID="cri-o://2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd" gracePeriod=25
Nov 02 07:38:58.186337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.186264    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:38:58.310216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310185    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\") pod \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") "
Nov 02 07:38:58.310441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310331    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\") pod \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") "
Nov 02 07:38:58.310441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310372    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\") pod \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") "
Nov 02 07:38:58.310441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310404    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\") pod \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\" (UID: \"e48d2be5-7a51-4cf5-816c-cd30f91d98ec\") "
Nov 02 07:38:58.310791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310759    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume" pod "e48d2be5-7a51-4cf5-816c-cd30f91d98ec" (UID: "e48d2be5-7a51-4cf5-816c-cd30f91d98ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:38:58.310882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.310796    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume" pod "e48d2be5-7a51-4cf5-816c-cd30f91d98ec" (UID: "e48d2be5-7a51-4cf5-816c-cd30f91d98ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:38:58.312591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.312557    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z" pod "e48d2be5-7a51-4cf5-816c-cd30f91d98ec" (UID: "e48d2be5-7a51-4cf5-816c-cd30f91d98ec"). InnerVolumeSpecName "kube-api-access-7x88z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:38:58.312686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.312662    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials" pod "e48d2be5-7a51-4cf5-816c-cd30f91d98ec" (UID: "e48d2be5-7a51-4cf5-816c-cd30f91d98ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:38:58.412197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.412161    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x88z\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-kube-api-access-7x88z\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:38:58.412197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.412191    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:38:58.412197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.412201    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:38:58.412468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.412209    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48d2be5-7a51-4cf5-816c-cd30f91d98ec-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:38:58.702792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.702760    4515 generic.go:356] "Generic (PLEG): container finished" podID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerID="2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd" exitCode=0
Nov 02 07:38:58.702953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.702813    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"
Nov 02 07:38:58.702953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.702847    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" event={"ID":"e48d2be5-7a51-4cf5-816c-cd30f91d98ec","Type":"ContainerDied","Data":"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"}
Nov 02 07:38:58.702953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.702931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898f6c747-89x6j" event={"ID":"e48d2be5-7a51-4cf5-816c-cd30f91d98ec","Type":"ContainerDied","Data":"328591a8975be9704084647819c20d43a265a94e8d4a8a2ee03c9515ec2596b7"}
Nov 02 07:38:58.703509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.702961    4515 scope.go:122] "RemoveContainer" containerID="2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"
Nov 02 07:38:58.712382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.712365    4515 scope.go:122] "RemoveContainer" containerID="2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"
Nov 02 07:38:58.712661 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:38:58.712643    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd\": container with ID starting with 2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd not found: ID does not exist" containerID="2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"
Nov 02 07:38:58.712724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.712669    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd"} err="failed to get container status \"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd\": rpc error: code = NotFound desc = could not find container \"2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd\": container with ID starting with 2b1873ad160e7be14dc74d51fd7f18203124bf33dc88f9c8722950b1f0398cbd not found: ID does not exist"
Nov 02 07:38:58.756377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.756350    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:38:58.762590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:58.762566    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898f6c747-89x6j"]
Nov 02 07:38:59.374439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:38:59.374405    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" path="/var/lib/kubelet/pods/e48d2be5-7a51-4cf5-816c-cd30f91d98ec/volumes"
Nov 02 07:39:04.677607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:04.677578    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:39:04.751156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:04.751126    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:39:07.089760 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.089723    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:39:07.090236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.090222    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing"
Nov 02 07:39:07.090236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.090236    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing"
Nov 02 07:39:07.090345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.090337    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48d2be5-7a51-4cf5-816c-cd30f91d98ec" containerName="payload-processing"
Nov 02 07:39:07.093547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.093530    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.112284 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.112255    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:39:07.138430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.138392    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a5e8106e020e\""
Nov 02 07:39:07.138430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.138401    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6751bb6a6774\""
Nov 02 07:39:07.174623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.174591    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.174806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.174629    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.174806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.174649    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8hf\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.174905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.174885    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.275864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.275824    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.275864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.275869    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.276144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.275887    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8hf\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.276144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.276001    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.276646 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.276620    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.276907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.276884    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.278482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.278461    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.299794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.299762    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8hf\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\") pod \"payload-processing-external-model-6859d58686-kckgm\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") " pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.406789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.406697    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:07.719270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.719235    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:39:07.817662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:07.817632    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" event={"ID":"2583944f-4bfa-48ee-878b-0c18fc8e6d3e","Type":"ContainerStarted","Data":"6f2ad841e72adaef7018ab17a030982d992dda76dd7b3569473da527804ddc6f"}
Nov 02 07:39:08.824277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:08.824242    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" event={"ID":"2583944f-4bfa-48ee-878b-0c18fc8e6d3e","Type":"ContainerStarted","Data":"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"}
Nov 02 07:39:08.824700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:08.824392    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:08.869074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:08.869023    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" podStartSLOduration=1.869008568 podStartE2EDuration="1.869008568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:08.866858192 +0000 UTC m=+893.629295100" watchObservedRunningTime="2026-11-02 07:39:08.869008568 +0000 UTC m=+893.631445471"
Nov 02 07:39:09.752406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:09.752360    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" podUID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" containerName="payload-processing" containerID="cri-o://db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db" gracePeriod=25
Nov 02 07:39:10.028264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.028239    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:39:10.114429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.114394    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\") pod \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") "
Nov 02 07:39:10.114590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.114446    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\") pod \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") "
Nov 02 07:39:10.114680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.114611    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\") pod \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") "
Nov 02 07:39:10.114847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.114817    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume" pod "0ddf0a98-9d5b-40e3-86da-56bace0575b5" (UID: "0ddf0a98-9d5b-40e3-86da-56bace0575b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:10.114977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.114860    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\") pod \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\" (UID: \"0ddf0a98-9d5b-40e3-86da-56bace0575b5\") "
Nov 02 07:39:10.115264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.115242    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume" pod "0ddf0a98-9d5b-40e3-86da-56bace0575b5" (UID: "0ddf0a98-9d5b-40e3-86da-56bace0575b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:10.115264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.115249    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:10.116980 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.116954    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc" pod "0ddf0a98-9d5b-40e3-86da-56bace0575b5" (UID: "0ddf0a98-9d5b-40e3-86da-56bace0575b5"). InnerVolumeSpecName "kube-api-access-jbppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:10.117068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.117053    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials" pod "0ddf0a98-9d5b-40e3-86da-56bace0575b5" (UID: "0ddf0a98-9d5b-40e3-86da-56bace0575b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:10.216754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.216723    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddf0a98-9d5b-40e3-86da-56bace0575b5-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:10.216754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.216752    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:10.216754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.216761    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbppc\" (UniqueName: \"kubernetes.io/projected/0ddf0a98-9d5b-40e3-86da-56bace0575b5-kube-api-access-jbppc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:10.836424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.836392    4515 generic.go:356] "Generic (PLEG): container finished" podID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" containerID="db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db" exitCode=0
Nov 02 07:39:10.836633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.836444    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"
Nov 02 07:39:10.836633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.836477    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" event={"ID":"0ddf0a98-9d5b-40e3-86da-56bace0575b5","Type":"ContainerDied","Data":"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"}
Nov 02 07:39:10.836633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.836532    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665db6c6c-slv9b" event={"ID":"0ddf0a98-9d5b-40e3-86da-56bace0575b5","Type":"ContainerDied","Data":"9b4c6795f55fada4702aa567a344da0cfb5bb770d12a6323529095fd4a3849a5"}
Nov 02 07:39:10.836633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.836554    4515 scope.go:122] "RemoveContainer" containerID="db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"
Nov 02 07:39:10.846959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.846933    4515 scope.go:122] "RemoveContainer" containerID="db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"
Nov 02 07:39:10.847259 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:39:10.847225    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db\": container with ID starting with db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db not found: ID does not exist" containerID="db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"
Nov 02 07:39:10.847377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.847271    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db"} err="failed to get container status \"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db\": rpc error: code = NotFound desc = could not find container \"db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db\": container with ID starting with db0beb657d7160bc4f9f27e452f894e058b47d6bf03b5d04f63cc624eaea58db not found: ID does not exist"
Nov 02 07:39:10.889260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.889226    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:39:10.901709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:10.901678    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-665db6c6c-slv9b"]
Nov 02 07:39:11.373616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.373583    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" path="/var/lib/kubelet/pods/0ddf0a98-9d5b-40e3-86da-56bace0575b5/volumes"
Nov 02 07:39:11.914708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.914671    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:39:11.915207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.915193    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" containerName="payload-processing"
Nov 02 07:39:11.915253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.915208    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" containerName="payload-processing"
Nov 02 07:39:11.915358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.915347    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddf0a98-9d5b-40e3-86da-56bace0575b5" containerName="payload-processing"
Nov 02 07:39:11.919981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.919962    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937234    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937526    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsf6\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937592    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937644    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937681    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937735    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937763    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937803    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.937932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.937878    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:11.941531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:11.941507    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:39:12.038506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038475    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038539    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038563    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsf6\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038595    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038638    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038670    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038900    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038937    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.038985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038977    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.039191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.038989    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.039191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.039138    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.039433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.039409    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.039796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.039750    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.041646 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.041625    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.041971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.041951    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.062992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.062964    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.063176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.063156    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsf6\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.235531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.235440    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:12.564176 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:39:12.564136    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09be35b7_df0b_45af_a266_609dbbdccdcc.slice/crio-31175583dd2ddb49a5493c7cc28a8862636113ebe03dc29a41c282ddb9755bf6 WatchSource:0}: Error finding container 31175583dd2ddb49a5493c7cc28a8862636113ebe03dc29a41c282ddb9755bf6: Status 404 returned error can't find the container with id 31175583dd2ddb49a5493c7cc28a8862636113ebe03dc29a41c282ddb9755bf6
Nov 02 07:39:12.565149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.565122    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:39:12.566494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.566461    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:12.566555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.566529    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:12.566591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.566561    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:12.851912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.851818    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" event={"ID":"09be35b7-df0b-45af-a266-609dbbdccdcc","Type":"ContainerStarted","Data":"cb4df52ea4c7c34728528c30aa4d5b95e556b5144beffb6405ce13fc37cb4951"}
Nov 02 07:39:12.851912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.851884    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" event={"ID":"09be35b7-df0b-45af-a266-609dbbdccdcc","Type":"ContainerStarted","Data":"31175583dd2ddb49a5493c7cc28a8862636113ebe03dc29a41c282ddb9755bf6"}
Nov 02 07:39:12.899459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:12.899403    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podStartSLOduration=1.899387345 podStartE2EDuration="1.899387345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:12.896528095 +0000 UTC m=+897.658965002" watchObservedRunningTime="2026-11-02 07:39:12.899387345 +0000 UTC m=+897.661824250"
Nov 02 07:39:13.236453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:13.236346    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:13.239344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:13.238842    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:13.239344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:13.238916    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:14.236385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:14.235944    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:14.236385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:14.236003    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:15.241778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.241740    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:15.242263 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.241803    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:39:15.242263 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.241816    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:15.344625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.343098    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:39:15.349343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.349295    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.376411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.376381    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:39:15.384298 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384265    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384298    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384462    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384626    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384668    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384692    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.384763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.384752    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4gs\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486257    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486320    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486367    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486410    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486449    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486524    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486564    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486785    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486838    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486873    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4gs\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.486991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486927    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.487043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.486986    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.487151 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.487127    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.487399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.487376    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.488743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.488717    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.488959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.488940    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.510778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.510743    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.510919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.510900    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4gs\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:15.666005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:15.665972    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:16.046968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.046498    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:39:16.048892 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:39:16.048858    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6d9467_bca4_4fe6_9880_688f88c66871.slice/crio-5eb85f2f736837c18f59018a5db2fd12bc38e744703db5a258028494e936adaa WatchSource:0}: Error finding container 5eb85f2f736837c18f59018a5db2fd12bc38e744703db5a258028494e936adaa: Status 404 returned error can't find the container with id 5eb85f2f736837c18f59018a5db2fd12bc38e744703db5a258028494e936adaa
Nov 02 07:39:16.051607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.051562    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.051696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.051664    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.051743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.051713    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.241331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.239598    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:16.241331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.239656    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:16.987339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.986340    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" event={"ID":"7d6d9467-bca4-4fe6-9880-688f88c66871","Type":"ContainerStarted","Data":"e2dd26835f35475bbcc115b46bc9a073d471b28225e189be119a006d44f5cb3d"}
Nov 02 07:39:16.987339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:16.986414    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" event={"ID":"7d6d9467-bca4-4fe6-9880-688f88c66871","Type":"ContainerStarted","Data":"5eb85f2f736837c18f59018a5db2fd12bc38e744703db5a258028494e936adaa"}
Nov 02 07:39:17.068335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.058682    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podStartSLOduration=2.058645262 podStartE2EDuration="2.058645262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:17.057162167 +0000 UTC m=+901.819599073" watchObservedRunningTime="2026-11-02 07:39:17.058645262 +0000 UTC m=+901.821082172"
Nov 02 07:39:17.240384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.239710    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:17.240384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.239775    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:17.672558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.671051    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:17.676333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.674146    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:17.676333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:17.674198    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:17.955718 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:39:17.955606    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:39:18.236821 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.236732    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:18.237428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.237394    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:18.712335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.687372    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:39:18.712335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.688061    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:18.712335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.688102    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:18.894330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:18.892470    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.238754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.238714    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:19.239346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.239291    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:19.669885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.669845    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:19.670210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.670166    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:19.685477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.685435    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.687643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.687594    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.688057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.688023    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:39:19.949079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:19.917089    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:39:20.213391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.206029    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:39:20.400730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.397296    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:39:20.454331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.453505    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 07:39:20.454331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.453568    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 07:39:20.697343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.691002    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:20.697343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:20.691064    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:21.666737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:21.666641    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:21.666737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:21.666704    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:21.811872 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:21.811844    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:21.812217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:21.812193    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:21.813171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:21.813146    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:39:22.674335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:22.669262    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:22.674335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:22.669361    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:23.666722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:23.666684    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 07:39:23.666906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:23.666743    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 07:39:24.674075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:24.674044    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:24.674562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:24.674431    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:24.675075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:24.675053    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:39:25.207129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.207078    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" podUID="e2eea6f5-aff5-4345-babb-61dcf06993e1" containerName="payload-processing" containerID="cri-o://d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c" gracePeriod=25
Nov 02 07:39:25.480728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.480702    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:39:25.568060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568027    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\") pod \"e2eea6f5-aff5-4345-babb-61dcf06993e1\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") "
Nov 02 07:39:25.568255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568087    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\") pod \"e2eea6f5-aff5-4345-babb-61dcf06993e1\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") "
Nov 02 07:39:25.568255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\") pod \"e2eea6f5-aff5-4345-babb-61dcf06993e1\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") "
Nov 02 07:39:25.568255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568159    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\") pod \"e2eea6f5-aff5-4345-babb-61dcf06993e1\" (UID: \"e2eea6f5-aff5-4345-babb-61dcf06993e1\") "
Nov 02 07:39:25.568547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568517    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume" pod "e2eea6f5-aff5-4345-babb-61dcf06993e1" (UID: "e2eea6f5-aff5-4345-babb-61dcf06993e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:25.568693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568665    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume" pod "e2eea6f5-aff5-4345-babb-61dcf06993e1" (UID: "e2eea6f5-aff5-4345-babb-61dcf06993e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:25.568755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.568688    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:25.570556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.570525    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt" pod "e2eea6f5-aff5-4345-babb-61dcf06993e1" (UID: "e2eea6f5-aff5-4345-babb-61dcf06993e1"). InnerVolumeSpecName "kube-api-access-tblrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:25.570650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.570576    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials" pod "e2eea6f5-aff5-4345-babb-61dcf06993e1" (UID: "e2eea6f5-aff5-4345-babb-61dcf06993e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:25.669439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.669409    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea6f5-aff5-4345-babb-61dcf06993e1-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:25.669439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.669438    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblrt\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-kube-api-access-tblrt\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:25.669439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:25.669447    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2eea6f5-aff5-4345-babb-61dcf06993e1-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:39:26.170458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.170425    4515 generic.go:356] "Generic (PLEG): container finished" podID="e2eea6f5-aff5-4345-babb-61dcf06993e1" containerID="d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c" exitCode=0
Nov 02 07:39:26.170930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.170476    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"
Nov 02 07:39:26.170930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.170510    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" event={"ID":"e2eea6f5-aff5-4345-babb-61dcf06993e1","Type":"ContainerDied","Data":"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"}
Nov 02 07:39:26.170930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.170566    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk" event={"ID":"e2eea6f5-aff5-4345-babb-61dcf06993e1","Type":"ContainerDied","Data":"53f1aeff441cb33c18940c1f4c77c61c52328dd1a3968c84b8b09baf9f940fda"}
Nov 02 07:39:26.170930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.170596    4515 scope.go:122] "RemoveContainer" containerID="d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"
Nov 02 07:39:26.180654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.180636    4515 scope.go:122] "RemoveContainer" containerID="d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"
Nov 02 07:39:26.180954 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:39:26.180935    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c\": container with ID starting with d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c not found: ID does not exist" containerID="d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"
Nov 02 07:39:26.181000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.180966    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c"} err="failed to get container status \"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c\": rpc error: code = NotFound desc = could not find container \"d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c\": container with ID starting with d795033b191ca8aacd2825be8b630e4e38143cbb65ec0da201f49306c718c26c not found: ID does not exist"
Nov 02 07:39:26.223114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.223076    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:39:26.235346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:26.235299    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-qhldk"]
Nov 02 07:39:27.378945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:27.378909    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2eea6f5-aff5-4345-babb-61dcf06993e1" path="/var/lib/kubelet/pods/e2eea6f5-aff5-4345-babb-61dcf06993e1/volumes"
Nov 02 07:39:30.395151 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.395114    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:39:30.406299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.406272    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e2eea6f5-aff5-4345-babb-61dcf06993e1" containerName="payload-processing"
Nov 02 07:39:30.406518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.406501    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2eea6f5-aff5-4345-babb-61dcf06993e1" containerName="payload-processing"
Nov 02 07:39:30.406759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.406737    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2eea6f5-aff5-4345-babb-61dcf06993e1" containerName="payload-processing"
Nov 02 07:39:30.434688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.434656    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.438560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.438527    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.445805    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.445984    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.446434    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.446513    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.446541    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2t4\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.446665    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.446765    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.447176    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.462369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.447504    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.630340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.621046    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.655549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.655448    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2t4\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674122    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674206    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674335    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674409    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674490    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674564    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.674593    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.675062    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.675665    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.717616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.684804    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.739648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.719289    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.742403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.742370    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.769021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.768984    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2t4\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.814334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.790640    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.832968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.831078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:30.855333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:30.850808    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:31.126903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:31.126544    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:31.601241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:31.588878    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.601241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:31.588979    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.601241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:31.589031    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.601241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:31.600681    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:39:32.281694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:32.274468    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" event={"ID":"0aafb588-87d9-49ef-9f5d-1c8d928a1db2","Type":"ContainerStarted","Data":"3fc1af314b6fd4bd0995f8528e9f898534f34ac1c2ee8824af314b064b4ad7bf"}
Nov 02 07:39:32.281694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:32.274581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" event={"ID":"0aafb588-87d9-49ef-9f5d-1c8d928a1db2","Type":"ContainerStarted","Data":"4ef9e16bc88f6e389463bfa801f5e80f5ae9c913866401fac1b8cb440ac3060c"}
Nov 02 07:39:32.358700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:32.358614    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podStartSLOduration=2.358592989 podStartE2EDuration="2.358592989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:32.338830014 +0000 UTC m=+917.101266921" watchObservedRunningTime="2026-11-02 07:39:32.358592989 +0000 UTC m=+917.121029896"
Nov 02 07:39:33.142189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:33.142146    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:33.152590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:33.152554    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:33.152892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:33.152862    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:34.137332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:34.131728    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:34.137332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:34.131795    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:35.140363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:35.139532    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:35.140363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:35.139594    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:36.131772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:36.131733    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:36.132053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:36.132024    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:37.132778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:37.132734    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:37.133420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:37.133387    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:38.136335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:38.132683    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:38.136335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:38.132757    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:39.127777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:39.127739    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 07:39:39.127990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:39.127821    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 07:39:40.131470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:40.131439    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:40.131942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:40.131719    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:40.132401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:40.132385    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:39:42.387552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:42.387521    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:39:45.635483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:45.635423    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:47.723082 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:39:47.723038    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:39:49.101402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.101372    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:39:49.106702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.106679    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.133138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.133036    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:39:49.139459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.139428    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:39:49.148895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.148867    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.166956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.166915    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:39:49.215730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215697    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.215730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215732    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.215948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215765    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.215948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215805    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.215948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215846    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215964    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.215990    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216020    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216124    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216171    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216237    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216287    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjn89\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216347    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216387    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216470    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216492    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.216513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216508    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.216620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.216593    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpknf\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.317615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317579    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.317816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317628    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjn89\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.317816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317646    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.317816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317761    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.317816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317799    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.317816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317819    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317842    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317912    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpknf\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317964    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.317979    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318011    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318036    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318075    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318152    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318187    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318212    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318261    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318294    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318339    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318361    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318376    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318602    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.318960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318661    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318862    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318936    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.318960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.318951    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.319234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.319111    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.319675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.319645    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.320582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.320552    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.320702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.320552    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.320849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.320829    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.320908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.320886    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.344977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.344942    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.345147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.345121    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjn89\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.345444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.345425    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-688jz\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.346538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.346515    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpknf\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.420982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.420897    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:49.468326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.468272    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:49.739392 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:39:49.739277    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe82cb21_73ce_4648_a095_cb44cd4b22ee.slice/crio-0dba2fb97035992fac51c54980b3fa67fbfb2d4d244b83345222c36f773d0289 WatchSource:0}: Error finding container 0dba2fb97035992fac51c54980b3fa67fbfb2d4d244b83345222c36f773d0289: Status 404 returned error can't find the container with id 0dba2fb97035992fac51c54980b3fa67fbfb2d4d244b83345222c36f773d0289
Nov 02 07:39:49.740615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.740590    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:39:49.741667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.741637    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.741756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.741702    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.741756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.741732    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.791323 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:39:49.791272    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23baa86_b864_4223_adc5_f8a7124cbe70.slice/crio-f75c08f16d80d753aeb2806f3370e435773cb98d82a68e1c53cb0b7205a236f3 WatchSource:0}: Error finding container f75c08f16d80d753aeb2806f3370e435773cb98d82a68e1c53cb0b7205a236f3: Status 404 returned error can't find the container with id f75c08f16d80d753aeb2806f3370e435773cb98d82a68e1c53cb0b7205a236f3
Nov 02 07:39:49.792272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.792248    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:39:49.793959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.793924    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.794046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.794002    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.794112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:49.794073    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:50.552244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.552200    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" event={"ID":"c23baa86-b864-4223-adc5-f8a7124cbe70","Type":"ContainerStarted","Data":"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"}
Nov 02 07:39:50.552925 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.552898    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" event={"ID":"c23baa86-b864-4223-adc5-f8a7124cbe70","Type":"ContainerStarted","Data":"f75c08f16d80d753aeb2806f3370e435773cb98d82a68e1c53cb0b7205a236f3"}
Nov 02 07:39:50.572341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.571650    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" event={"ID":"be82cb21-73ce-4648-a095-cb44cd4b22ee","Type":"ContainerStarted","Data":"8c2d38a8bdf6c523ba649aa6e7b355c15a0d2f8fe164547acf916d434cf2b752"}
Nov 02 07:39:50.572341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.571741    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" event={"ID":"be82cb21-73ce-4648-a095-cb44cd4b22ee","Type":"ContainerStarted","Data":"0dba2fb97035992fac51c54980b3fa67fbfb2d4d244b83345222c36f773d0289"}
Nov 02 07:39:50.668562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.668485    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podStartSLOduration=1.66846435 podStartE2EDuration="1.66846435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:50.612262552 +0000 UTC m=+935.374699459" watchObservedRunningTime="2026-11-02 07:39:50.66846435 +0000 UTC m=+935.430901258"
Nov 02 07:39:50.681823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:50.681771    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podStartSLOduration=1.6817557650000001 podStartE2EDuration="1.681755765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:50.679646588 +0000 UTC m=+935.442083494" watchObservedRunningTime="2026-11-02 07:39:50.681755765 +0000 UTC m=+935.444192712"
Nov 02 07:39:51.470350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.464376    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:51.470350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.466492    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:51.470350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.466549    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:51.470350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.469696    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:51.470736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.470570    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:51.470736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:51.470616    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:52.427335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:52.424660    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:52.427335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:52.424722    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:52.472856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:52.472715    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:52.472856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:52.472775    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:53.429334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:53.426643    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:53.429334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:53.426703    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:53.470336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:53.470269    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:53.470550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:53.470359    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:54.428335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:54.424732    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:54.428335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:54.424810    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:54.469338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:54.468729    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:54.469338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:54.468798    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:55.433204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:55.432721    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:55.433204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:55.432793    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:55.468743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:55.468713    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:55.468969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:55.468944    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:56.422268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:56.422233    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-688jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 07:39:56.422763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:56.422300    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 07:39:56.469250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:56.469212    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 07:39:56.469462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:56.469269    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 07:39:57.426836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.426800    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:57.427488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.427463    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:57.428658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.428636    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:39:57.472984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.472952    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:57.473252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.473233    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:57.473846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:57.473828    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:39:59.390479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.390438    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:39:59.395853 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.395826    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.432217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.432181    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:39:59.566121 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.566077    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcvn\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.566497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.566472    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.566693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.566678    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.566871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.566856    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.567021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.567008    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.567177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.567161    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.567399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.567381    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.567588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.567570    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.567742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.567728    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.669105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.669017    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.670390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.670155    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.670695 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.670674    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcvn\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.670876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.670849    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.671536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.671516    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.671796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.671778    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.671942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.671928    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.672091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.672077    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.672275 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.672182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.672620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.672513    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.672876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.672768    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.673244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.673229    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.673832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.673807    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.674552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.674533    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.675816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.675799    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.677069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.677029    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.871385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.870563    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcvn\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:39:59.871385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:39:59.871265    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:00.015344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.014491    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:00.843763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.843730    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:40:00.847039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.846961    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:00.847211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.847152    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:00.847300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.847278    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:00.858631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.858594    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:00.938406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.938363    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:40:00.961343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:00.944395    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.015934    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016049    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016166    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllnn\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016199    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016251    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.018485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016278    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.046886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.016302    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.046886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.044495    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.046886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.044535    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.180031    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllnn\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.180106    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181088    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181146    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181179    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181215    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.181348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181246    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.186356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.181868    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.186356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.182565    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.186915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.186868    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.196332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.187288    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.197454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.197429    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.198148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.197993    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.201459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.201435    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.227275 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.226670    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.269382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.263442    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllnn\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.269382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.264065    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.285344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.284966    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:01.518266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:01.517872    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:02.411342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.407399    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.430353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.421423    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" event={"ID":"b349a217-1c19-44b1-a62c-4d25fea8bd10","Type":"ContainerStarted","Data":"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"}
Nov 02 07:40:02.430353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.421509    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" event={"ID":"b349a217-1c19-44b1-a62c-4d25fea8bd10","Type":"ContainerStarted","Data":"d48b0232665a437e2eef731cd02916b37e65e1ab4ca885e401f5020d80b0815b"}
Nov 02 07:40:02.682396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.609546    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podStartSLOduration=3.6095263060000002 podStartE2EDuration="3.609526306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:02.513009942 +0000 UTC m=+947.275446861" watchObservedRunningTime="2026-11-02 07:40:02.609526306 +0000 UTC m=+947.371963214"
Nov 02 07:40:02.705463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.624544    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.726415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.691393    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.726415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.708526    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:40:02.869003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.854462    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.884333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.880745    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:02.884640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.884601    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:02.884845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:02.884822    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:03.018363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.017873    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:03.018363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.018013    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:03.018363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.018064    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:03.676988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.676543    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:40:03.742523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.742433    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" event={"ID":"33b98d21-89e3-4b52-838b-11e3f86ac7a3","Type":"ContainerStarted","Data":"99b819b4926f73d0568691fe441ac1b203c0eaceb650beafed2911799889620f"}
Nov 02 07:40:03.776406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.765499    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:03.776406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.765835    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:40:03.776406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.766242    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:03.910395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.865638    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:03.931422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.920623    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:03.998662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:03.998586    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:04.021657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:04.021623    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:04.021912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:04.021882    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:04.058344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:04.048267    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:04.765032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:04.764977    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" event={"ID":"33b98d21-89e3-4b52-838b-11e3f86ac7a3","Type":"ContainerStarted","Data":"e820af28d57fe22038cf1c8fae7da82d03200b2a7d53b4f25f266b733c6e0675"}
Nov 02 07:40:04.854333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:04.853258    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podStartSLOduration=4.853239796 podStartE2EDuration="4.853239796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:04.83480301 +0000 UTC m=+949.597239918" watchObservedRunningTime="2026-11-02 07:40:04.853239796 +0000 UTC m=+949.615676704"
Nov 02 07:40:05.015536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:05.015452    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:05.015536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:05.015501    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:05.530362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:05.521475    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:05.538109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:05.532809    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:05.538109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:05.532972    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:06.015753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:06.015717    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:06.016137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:06.015773    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:06.524333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:06.520661    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:06.524333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:06.520719    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:07.015134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:07.015094    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:07.015357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:07.015149    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:07.535332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:07.533998    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:07.535332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:07.534082    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:08.015021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:08.014986    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:08.015201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:08.015054    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:08.531857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:08.531821    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:08.532207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:08.532160    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:09.015540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:09.015505    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 07:40:09.015945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:09.015569    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 07:40:09.521727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:09.521689    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:09.522002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:09.521975    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:10.130433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:10.128661    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:10.130433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:10.130246    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:10.133700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:10.133652    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:40:10.519269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:10.519227    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 07:40:10.519515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:10.519298    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 07:40:11.464123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:11.464090    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:40:11.627928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:11.627896    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:11.629190 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:11.628562    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:11.630387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:11.630365    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:40:16.465350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.465265    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" podUID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" containerName="payload-processing" containerID="cri-o://9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17" gracePeriod=25
Nov 02 07:40:16.745725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.745699    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:40:16.787978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.787938    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\") pod \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") "
Nov 02 07:40:16.788203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.788112    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\") pod \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") "
Nov 02 07:40:16.788283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.788227    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\") pod \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") "
Nov 02 07:40:16.788375 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.788298    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\") pod \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\" (UID: \"2583944f-4bfa-48ee-878b-0c18fc8e6d3e\") "
Nov 02 07:40:16.789099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.789063    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume" pod "2583944f-4bfa-48ee-878b-0c18fc8e6d3e" (UID: "2583944f-4bfa-48ee-878b-0c18fc8e6d3e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:16.789233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.789152    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume" pod "2583944f-4bfa-48ee-878b-0c18fc8e6d3e" (UID: "2583944f-4bfa-48ee-878b-0c18fc8e6d3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:16.790703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.790658    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials" pod "2583944f-4bfa-48ee-878b-0c18fc8e6d3e" (UID: "2583944f-4bfa-48ee-878b-0c18fc8e6d3e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:16.791681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.791656    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf" pod "2583944f-4bfa-48ee-878b-0c18fc8e6d3e" (UID: "2583944f-4bfa-48ee-878b-0c18fc8e6d3e"). InnerVolumeSpecName "kube-api-access-4b8hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:16.865848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.865815    4515 generic.go:356] "Generic (PLEG): container finished" podID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" containerID="9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17" exitCode=0
Nov 02 07:40:16.865990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.865858    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm"
Nov 02 07:40:16.865990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.865873    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" event={"ID":"2583944f-4bfa-48ee-878b-0c18fc8e6d3e","Type":"ContainerDied","Data":"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"}
Nov 02 07:40:16.865990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.865939    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6859d58686-kckgm" event={"ID":"2583944f-4bfa-48ee-878b-0c18fc8e6d3e","Type":"ContainerDied","Data":"6f2ad841e72adaef7018ab17a030982d992dda76dd7b3569473da527804ddc6f"}
Nov 02 07:40:16.865990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.865960    4515 scope.go:122] "RemoveContainer" containerID="9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"
Nov 02 07:40:16.876171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.876150    4515 scope.go:122] "RemoveContainer" containerID="9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"
Nov 02 07:40:16.876504 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:40:16.876483    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17\": container with ID starting with 9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17 not found: ID does not exist" containerID="9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"
Nov 02 07:40:16.876570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.876512    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17"} err="failed to get container status \"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17\": rpc error: code = NotFound desc = could not find container \"9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17\": container with ID starting with 9576ab241b4cc7677eaa9eb5f93b461d01a2978def8e54d808a73e9530956c17 not found: ID does not exist"
Nov 02 07:40:16.889776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.889753    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:40:16.889776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.889777    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:40:16.889919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.889788    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8hf\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-kube-api-access-4b8hf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:40:16.889919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.889796    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2583944f-4bfa-48ee-878b-0c18fc8e6d3e-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:40:16.919323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.919283    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:40:16.924681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:16.924648    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6859d58686-kckgm"]
Nov 02 07:40:17.373095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:17.373065    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" path="/var/lib/kubelet/pods/2583944f-4bfa-48ee-878b-0c18fc8e6d3e/volumes"
Nov 02 07:40:17.725466 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:40:17.725369    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:40:19.212392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.212361    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 07:40:19.212891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.212879    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" containerName="payload-processing"
Nov 02 07:40:19.212891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.212889    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" containerName="payload-processing"
Nov 02 07:40:19.213013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.212985    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2583944f-4bfa-48ee-878b-0c18fc8e6d3e" containerName="payload-processing"
Nov 02 07:40:19.217656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.217634    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.253602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.253565    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 07:40:19.326680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326646    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.326680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326683    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.326908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326702    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.326908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326859    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.327007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326912    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmj6\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.327007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.326970    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.327007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.327000    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.327103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.327090    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.327133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.327121    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.331539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.331505    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:40:19.428472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428440    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428482    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428542    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428560    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428647    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428717    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmj6\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428747    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428773    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.428914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428870    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.429113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.428923    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.429414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.429390    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.429532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.429399    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.429602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.429526    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.431251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.431225    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.431380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.431275    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.452062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.452031    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.452205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.452185    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmj6\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.533089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.533058    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:19.850942 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:40:19.850912    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94421cd_cc31_4922_95fd_06d462d95d38.slice/crio-ea683a523dc805316d0a690c140125d384b197c499a6d9dfaf72e1c1cebad519 WatchSource:0}: Error finding container ea683a523dc805316d0a690c140125d384b197c499a6d9dfaf72e1c1cebad519: Status 404 returned error can't find the container with id ea683a523dc805316d0a690c140125d384b197c499a6d9dfaf72e1c1cebad519
Nov 02 07:40:19.851808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.851780    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 07:40:19.853516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.853485    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:19.853596 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.853546    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:19.853596 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:19.853574    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:20.003603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:20.003562    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" event={"ID":"c94421cd-cc31-4922-95fd-06d462d95d38","Type":"ContainerStarted","Data":"ea683a523dc805316d0a690c140125d384b197c499a6d9dfaf72e1c1cebad519"}
Nov 02 07:40:21.050335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:21.039534    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" event={"ID":"c94421cd-cc31-4922-95fd-06d462d95d38","Type":"ContainerStarted","Data":"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"}
Nov 02 07:40:21.123336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:21.123103    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podStartSLOduration=2.123079459 podStartE2EDuration="2.123079459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:21.105112931 +0000 UTC m=+965.867549838" watchObservedRunningTime="2026-11-02 07:40:21.123079459 +0000 UTC m=+965.885516366"
Nov 02 07:40:21.535576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:21.535539    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:21.535935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:21.535911    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:21.536083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:21.536061    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:22.542335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:22.539558    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:22.542335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:22.539619    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:22.688452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:22.688392    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.033332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.028430    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.093335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.090425    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.421392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.420427    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.541334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.540440    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:23.541334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.540526    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:23.727775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.724963    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.727775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.725377    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.727775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.725966    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:40:23.760462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:23.760418    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 02 07:40:24.044995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.044937    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.046104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.046082    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:24.096506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.096426    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.097510 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.097486    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:24.433502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.433392    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.461803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.461775    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:40:24.772490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:24.772443    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.012335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.012173    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:40:25.038663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.033855    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:25.038663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.033927    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:25.101343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.097540    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 07:40:25.104742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.104613    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.574340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.573537    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:25.574340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.573611    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:25.820210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.820167    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.850296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.850201    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:25.882211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.855795    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 02 07:40:25.882731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.867801    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:25.897991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:25.886215    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:40:26.548737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:26.548703    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:26.549066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:26.549022    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:27.539700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:27.539661    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:27.540325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:27.540274    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:28.539735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:28.539693    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:28.540335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:28.540285    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:29.533928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:29.533884    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:40:29.534132 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:29.534014    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 07:40:30.957116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:30.957080    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:30.957618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:30.957597    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:30.958323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:30.958292    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 07:40:40.812334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:40.751438    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:40.812334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:40.809393    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:40.907348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:40.881819    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.344392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.243755    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.344392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.256599    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:40:41.344392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.286447    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.344392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.337197    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:41.344392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.337281    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.766402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.751986    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:40:41.766402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.753600    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:40:41.816337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.815916    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:40:41.948361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.920720    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.950413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.950384    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:40:41.950739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:41.950724    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:42.258517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.258471    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:40:42.315296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.315265    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:40:42.316743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.315645    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.321333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.317170    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:42.722823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.722720    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:42.759337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.758799    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.759337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.757522    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:40:42.817548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.816211    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:40:42.955547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:42.955499    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.317843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.317786    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.332667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.331966    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.363876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.363844    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:40:43.537206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.537163    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:43.538432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.538399    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.682911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.682823    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:40:43.817592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.817534    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.900146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.900112    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:40:43.903257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.902661    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:40:43.956241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:43.955284    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:40:44.523783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.523748    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:40:44.528351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.528324    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.553491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.553462    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:40:44.556578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556545    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556606    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556646    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556673    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556708    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556741    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556764    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556798    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.556972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.556843    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpfq\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.657722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657681    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.657922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657743    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.657922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657780    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.657922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657804    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.657922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657838    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.657962    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658014    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658056    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658128    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpfq\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658244    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658297    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.658715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.658691    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.659035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.659015    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.660224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.660202    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.660742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.660720    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.681952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.681931    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpfq\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.682052 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.681955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:44.846944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:44.846621    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:45.206836 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:40:45.206801    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46284b42_5f56_41c0_a1d3_0489a5929b64.slice/crio-7dd62936b4a4c082aa8e76cf5c2cc08af3f64178ed09144eee5ab9d4076d26f5 WatchSource:0}: Error finding container 7dd62936b4a4c082aa8e76cf5c2cc08af3f64178ed09144eee5ab9d4076d26f5: Status 404 returned error can't find the container with id 7dd62936b4a4c082aa8e76cf5c2cc08af3f64178ed09144eee5ab9d4076d26f5
Nov 02 07:40:45.208266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.208240    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:40:45.209349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.209302    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.209456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.209393    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.209456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.209447    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.654639 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.654587    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" event={"ID":"46284b42-5f56-41c0-a1d3-0489a5929b64","Type":"ContainerStarted","Data":"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"}
Nov 02 07:40:45.654848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.654686    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" event={"ID":"46284b42-5f56-41c0-a1d3-0489a5929b64","Type":"ContainerStarted","Data":"7dd62936b4a4c082aa8e76cf5c2cc08af3f64178ed09144eee5ab9d4076d26f5"}
Nov 02 07:40:45.724840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.724773    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podStartSLOduration=1.724753927 podStartE2EDuration="1.724753927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:45.721231251 +0000 UTC m=+990.483668158" watchObservedRunningTime="2026-11-02 07:40:45.724753927 +0000 UTC m=+990.487190834"
Nov 02 07:40:45.886333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:45.873474    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:46.849023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:46.848988    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:46.849201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:46.849042    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:47.751571 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:40:47.751531    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:40:47.848386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:47.848348    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:47.848566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:47.848413    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:48.848899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:48.848862    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:48.849303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:48.848933    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:49.144387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.144160    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:40:49.150216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.150190    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.150934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.150910    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:40:49.165360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.165271    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-77dc65fb\""
Nov 02 07:40:49.177673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.177648    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-77dc65fb-dockercfg-44vb9\""
Nov 02 07:40:49.208589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.208528    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:40:49.229332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.215034    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.286661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.280181    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.286661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.280278    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw92k\" (UniqueName: \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.308400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.308353    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:40:49.412416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.390602    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmq2\" (UniqueName: \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.412416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.390685    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.412416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.390769    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw92k\" (UniqueName: \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.412416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.390961    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.465713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.465676    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.500343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.498337    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw92k\" (UniqueName: \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.500794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.500768    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:49.509350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.501031    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.509350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.501114    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmq2\" (UniqueName: \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.509350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.502471    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.547839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.547810    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmq2\" (UniqueName: \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.612056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.609434    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:49.855660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.855625    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:49.861422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.857791    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:49.861422 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:40:49.860942    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5254b663_d289_4fc1_abcf_ebd589039ee9.slice/crio-2327b010a71cb986db37a1e65b006b902af41ef51097f5f089af8a9f91c177dc WatchSource:0}: Error finding container 2327b010a71cb986db37a1e65b006b902af41ef51097f5f089af8a9f91c177dc: Status 404 returned error can't find the container with id 2327b010a71cb986db37a1e65b006b902af41ef51097f5f089af8a9f91c177dc
Nov 02 07:40:49.861687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.861668    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:40:49.979339 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:40:49.979274    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0970f1e5_5b88_44b2_a9dd_8f6266da5a34.slice/crio-75a0c156ec1cb979a25f733f3842c08c9a26e073155b4a4fb8f9e36112c03bb2 WatchSource:0}: Error finding container 75a0c156ec1cb979a25f733f3842c08c9a26e073155b4a4fb8f9e36112c03bb2: Status 404 returned error can't find the container with id 75a0c156ec1cb979a25f733f3842c08c9a26e073155b4a4fb8f9e36112c03bb2
Nov 02 07:40:49.980013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:49.979987    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:40:50.689363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.689321    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" event={"ID":"5254b663-d289-4fc1-abcf-ebd589039ee9","Type":"ContainerStarted","Data":"7f79c00916bb7f00f7dedf5b1658ce4350ef8acb9f57ef634366e7c65c842c4b"}
Nov 02 07:40:50.689561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.689431    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" event={"ID":"5254b663-d289-4fc1-abcf-ebd589039ee9","Type":"ContainerStarted","Data":"2327b010a71cb986db37a1e65b006b902af41ef51097f5f089af8a9f91c177dc"}
Nov 02 07:40:50.689561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.689472    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:40:50.691159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.691129    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" event={"ID":"0970f1e5-5b88-44b2-a9dd-8f6266da5a34","Type":"ContainerStarted","Data":"38a8b0de284adbf5820cd75a5c348f8da34d1e381ba65c9c65ab4a482c3e1d4f"}
Nov 02 07:40:50.691285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.691183    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" event={"ID":"0970f1e5-5b88-44b2-a9dd-8f6266da5a34","Type":"ContainerStarted","Data":"75a0c156ec1cb979a25f733f3842c08c9a26e073155b4a4fb8f9e36112c03bb2"}
Nov 02 07:40:50.691285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.691213    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:40:50.743074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.743003    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" podStartSLOduration=1.742981559 podStartE2EDuration="1.742981559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:50.736786499 +0000 UTC m=+995.499223409" watchObservedRunningTime="2026-11-02 07:40:50.742981559 +0000 UTC m=+995.505418466"
Nov 02 07:40:50.779733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.779669    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" podStartSLOduration=1.779653672 podStartE2EDuration="1.779653672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:50.777181687 +0000 UTC m=+995.539618592" watchObservedRunningTime="2026-11-02 07:40:50.779653672 +0000 UTC m=+995.542090578"
Nov 02 07:40:50.848620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.848581    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:50.848795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:50.848642    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:51.848931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:51.848891    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 07:40:51.849362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:51.848960    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 07:40:52.852981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:52.852951    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:52.853389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:52.853215    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:52.853984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:52.853969    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:40:57.213277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.213243    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:40:57.217249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.217225    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.248505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.248470    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:40:57.261929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.261900    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:40:57.265888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.265864    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.284245 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.284215    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-77dc65fb-serving-cert\""
Nov 02 07:40:57.296093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.289383    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:40:57.362339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362287    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.362529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362398    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362428    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362534    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.362629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362589    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhb8\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362619    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.362740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362712    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362749    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vvz\" (UniqueName: \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.362780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362768    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362853 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362904 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362892    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.362952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.362938    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.363027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.363007    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470604    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470686    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470737    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470870    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470915    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.470944    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471015    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhb8\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471145    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471221    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471281    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vvz\" (UniqueName: \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.471331    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.473043    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.474188    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.479413    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.480460    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.482368    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.482558    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.487379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.482964    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.511967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.509368    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.525058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.523879    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.525058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.524950    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.552172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.526061    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vvz\" (UniqueName: \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.564593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.564284    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.564593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.564458    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhb8\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.564593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.564535    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.596058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.583549    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:57.831865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.831832    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:57.913818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.912200    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:40:57.915456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:57.915435    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:40:58.140157 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:40:58.140124    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc909f7c3_a340_4e1c_b15a_ffe6132e9835.slice/crio-23c5712f7ddbfe4af39e8f1ad92fa74fcc3bbac53e248a3a3ec0db7d01941408 WatchSource:0}: Error finding container 23c5712f7ddbfe4af39e8f1ad92fa74fcc3bbac53e248a3a3ec0db7d01941408: Status 404 returned error can't find the container with id 23c5712f7ddbfe4af39e8f1ad92fa74fcc3bbac53e248a3a3ec0db7d01941408
Nov 02 07:40:58.140675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.140653    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:40:58.142875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.142842    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:58.143026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.142919    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:58.143026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.142972    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:58.743540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.743444    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" event={"ID":"9410f552-92c8-41dd-8833-83f9c7a99f1f","Type":"ContainerStarted","Data":"78ccfea6b6471d8735c86673acfd98ea46fe2bb69dbb0ce9fd79ff15fb029e2e"}
Nov 02 07:40:58.744554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.744457    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" event={"ID":"9410f552-92c8-41dd-8833-83f9c7a99f1f","Type":"ContainerStarted","Data":"370108d416b4dc5fbe4defba678c10eb9603c130c424befd60fe4686a71bcf01"}
Nov 02 07:40:58.744705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.744580    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:40:58.746953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.746924    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" event={"ID":"c909f7c3-a340-4e1c-b15a-ffe6132e9835","Type":"ContainerStarted","Data":"78c476639f03add6e6cb05614e3923261c0f9bcc608afd13d0126c5b9a3c17e2"}
Nov 02 07:40:58.747096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.746975    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" event={"ID":"c909f7c3-a340-4e1c-b15a-ffe6132e9835","Type":"ContainerStarted","Data":"23c5712f7ddbfe4af39e8f1ad92fa74fcc3bbac53e248a3a3ec0db7d01941408"}
Nov 02 07:40:58.789079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.789029    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" podStartSLOduration=1.328296657 podStartE2EDuration="1.789015718s" totalImagesPullingTime="460.719061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:58.786523224 +0000 UTC m=+1003.548960129" watchObservedRunningTime="2026-11-02 07:40:58.789015718 +0000 UTC m=+1003.551452624"
Nov 02 07:40:58.832192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.832161    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:40:58.835702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:58.835656    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podStartSLOduration=1.8356427800000001 podStartE2EDuration="1.83564278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:58.832458877 +0000 UTC m=+1003.594895783" watchObservedRunningTime="2026-11-02 07:40:58.83564278 +0000 UTC m=+1003.598079685"
Nov 02 07:40:59.684568 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:59.684513    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:40:59.837337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:59.837019    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:40:59.837337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:40:59.837134    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:00.844333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:00.843756    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:00.844333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:00.843817    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:01.324338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.324155    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:01.345334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.337953    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:01.383336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.382509    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:01.827334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.785521    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:41:01.827334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.805519    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:41:01.851585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.829136    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:41:01.865604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.865558    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:01.865809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:01.865640    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:02.351335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.328413    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.371339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.331393    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:41:02.371339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.340400    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:41:02.387331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.386506    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 07:41:02.449665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.449618    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.481334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.477252    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:41:02.481334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.477366    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:41:02.481334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.477410    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:41:02.481334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.477499    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:41:02.507337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.482583    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.564930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.564896    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d47676-openshift-default-dockercfg-hcl98\""
Nov 02 07:41:02.632745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.632654    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.636486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.636450    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdl2d\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.724344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.699541    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.725334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.725285    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.805331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.726333    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.825066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.814566    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.836875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.836846    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.837229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.837206    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.837473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.837453    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:02.838231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.838195    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:02.838400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.836725    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.860346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.838395    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:02.879513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.867716    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:41:02.966636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:02.966527    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.037410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.020397    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.037807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.037767    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.038301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.038161    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.038491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.038471    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.038958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.038741    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.061345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.050731    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.086336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.061624    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.086732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.086696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.086963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.086937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdl2d\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.087181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.087164    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.099340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.062778    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.236386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.066246    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.236386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.095585    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.279475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.279427    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.307343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.296208    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.422335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.387644    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:03.422335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.398115    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.422335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.398335    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.469334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.450782    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:03.659335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.642214    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdl2d\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.796334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.765481    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\") pod \"e2e-api-a-d47676-openshift-default-6df488c785-gm487\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") " pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:03.805334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.796928    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 02 07:41:03.855503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:03.855458    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.109331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.020787    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:04.131219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.131174    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:04.133350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.133296    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:04.177336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.157853    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:41:04.528728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.528683    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.550373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.529190    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.550796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.529357    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.575859 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.569610    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.868508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.868172    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.880404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.874161    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:04.880404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:04.874220    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:05.583752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.583704    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:05.615616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.613547    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:05.615616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.613827    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:41:05.615616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.614070    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:41:05.716078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.716050    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:41:05.836512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.835460    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:05.837544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.837514    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:05.837999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.837206    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:41:05.899572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:05.899523    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:41:06.121353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.113248    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:06.121353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.113386    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:06.121353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.113434    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:06.159478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.159395    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:41:06.525750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.525688    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:41:06.598495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.598423    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:06.599762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.599722    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:41:06.601361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.601327    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:41:06.777536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.777446    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:41:06.845905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.845428    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:06.845905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:06.845489    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:07.116396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:07.116054    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerStarted","Data":"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"}
Nov 02 07:41:07.116396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:07.116135    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerStarted","Data":"9b7eb745666a197639fd4d9eddc061db3a77e0c8b5d17c3520ae3c7daea3730e"}
Nov 02 07:41:07.167876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:07.167823    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podStartSLOduration=6.167809929 podStartE2EDuration="6.167809929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:07.164931443 +0000 UTC m=+1011.927368350" watchObservedRunningTime="2026-11-02 07:41:07.167809929 +0000 UTC m=+1011.930246918"
Nov 02 07:41:07.836336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:07.832818    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:07.836336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:07.832898    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:08.042418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.038666    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:08.048186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.048154    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:08.143446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.143347    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:08.144647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.144625    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:41:08.832810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.832768    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:41:08.832962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:08.832835    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 07:41:10.328731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.328692    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:41:10.334183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.334157    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.368732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.365219    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:41:10.369142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.369116    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fe085e-openshift-default-dockercfg-sshkk\""
Nov 02 07:41:10.410850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.410819    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.410869    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.410985    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411048    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411082    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411107    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411159    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411237    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.411274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.411262    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8sqw\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520462    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520540    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520580    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520623    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520649    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520671    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520720    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520793    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.520819    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8sqw\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.521983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.521947    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.525331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.522638    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.525331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.522905    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.525331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.523163    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.525331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.523412    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.543432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.539668    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.553960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.553926    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.569280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.569248    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.648502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.638860    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8sqw\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\") pod \"e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") " pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.659679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.659637    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:10.883964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.883929    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:10.884206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:10.884186    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:11.166139 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:11.165302    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fc34ba_3dc6_4c8d_9dbb_b62a18064652.slice/crio-20c4ac1587c074e28ced08114bdfee7473a7dcbb939710c58d626bdfd8045197 WatchSource:0}: Error finding container 20c4ac1587c074e28ced08114bdfee7473a7dcbb939710c58d626bdfd8045197: Status 404 returned error can't find the container with id 20c4ac1587c074e28ced08114bdfee7473a7dcbb939710c58d626bdfd8045197
Nov 02 07:41:11.166139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.165833    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:41:11.170276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.169690    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:11.170276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.169784    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:11.170276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.169836    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:11.670380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.670346    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:41:11.670909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.670751    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:41:11.671475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.671457    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:41:11.685255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.685221    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:41:11.689464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.689445    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.705516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.705481    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d47676\""
Nov 02 07:41:11.708511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.708485    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d47676-dockercfg-9qhvm\""
Nov 02 07:41:11.710540 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.710511    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:41:11.776031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.762214    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:41:11.776031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.767957    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7zj\" (UniqueName: \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.776031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.769038    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.822333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.815591    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:11.887337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.874191    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.887578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.887546    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.887794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.887770    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:11.888017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.887998    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7zj\" (UniqueName: \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:11.897338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:11.888209    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbxp\" (UniqueName: \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.014535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.005462    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbxp\" (UniqueName: \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.015111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.015035    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.016653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.016612    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.043016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.042984    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:41:12.103625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.103562    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7zj\" (UniqueName: \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\") pod \"payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:12.108942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.106586    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbxp\" (UniqueName: \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\") pod \"payload-processing-e2e-api-a-d47676-668589f87f-kz8h4\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.109663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.109636    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:12.184208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.179344    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerStarted","Data":"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"}
Nov 02 07:41:12.184208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.179407    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerStarted","Data":"20c4ac1587c074e28ced08114bdfee7473a7dcbb939710c58d626bdfd8045197"}
Nov 02 07:41:12.229819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.229708    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podStartSLOduration=2.229687805 podStartE2EDuration="2.229687805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:12.224778018 +0000 UTC m=+1016.987214924" watchObservedRunningTime="2026-11-02 07:41:12.229687805 +0000 UTC m=+1016.992124713"
Nov 02 07:41:12.262813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.262779    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:41:12.269220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.269189    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.288029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.287995    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:41:12.305793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.305759    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:12.320959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.320926    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.320978    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cfs\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321029    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321088    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321122    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321265    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321330    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321364    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.321477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.321403    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.436729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.436663    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.437009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.436977    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.437189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.437159    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cfs\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.437930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.437906    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.438177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.438161    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.438671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.438643    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.439232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.438570    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.439551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.439533    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.440475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.440251    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.440965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.440903    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.441097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.441079    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.442095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.442035    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.444118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.443456    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.444118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.443526    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.447089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.447068    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.447290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.447271    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.489627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.489550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cfs\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.491231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.491200    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.508271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.507601    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:41:12.510580 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:12.510552    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7da7301_9856_46dc_93de_58745aab4e7c.slice/crio-ed9339ecc803d1dc1ea0faf2273bc89c21c6c76b578966d3e4039e5786f7f4f1 WatchSource:0}: Error finding container ed9339ecc803d1dc1ea0faf2273bc89c21c6c76b578966d3e4039e5786f7f4f1: Status 404 returned error can't find the container with id ed9339ecc803d1dc1ea0faf2273bc89c21c6c76b578966d3e4039e5786f7f4f1
Nov 02 07:41:12.590905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.590872    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:12.660899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.660867    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:12.666665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.666637    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:12.755757 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:12.755722    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f869533_f3ef_4491_8275_517bb9d247d7.slice/crio-b693d7901fb06d22bf8fddf1d24bfd35fd55ac199b5a88a3656e0a00d9a586ff WatchSource:0}: Error finding container b693d7901fb06d22bf8fddf1d24bfd35fd55ac199b5a88a3656e0a00d9a586ff: Status 404 returned error can't find the container with id b693d7901fb06d22bf8fddf1d24bfd35fd55ac199b5a88a3656e0a00d9a586ff
Nov 02 07:41:12.757432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.757404    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:41:12.926300 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:12.926266    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbbb7ea_71e6_4ffd_8feb_31efd80d4d5c.slice/crio-c8ad0cbee5bf1d32eb71d1f45796b070880d702fe931e5c62b6b202d04d12e8e WatchSource:0}: Error finding container c8ad0cbee5bf1d32eb71d1f45796b070880d702fe931e5c62b6b202d04d12e8e: Status 404 returned error can't find the container with id c8ad0cbee5bf1d32eb71d1f45796b070880d702fe931e5c62b6b202d04d12e8e
Nov 02 07:41:12.927609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.927584    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:41:12.928868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.928836    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:12.928949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.928901    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:12.928949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:12.928935    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:13.213388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.213075    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" event={"ID":"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c","Type":"ContainerStarted","Data":"26a744d8cb119a4075a7a18a42010c1c8b951d0b60d8c2926665de6384533594"}
Nov 02 07:41:13.213388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.213144    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" event={"ID":"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c","Type":"ContainerStarted","Data":"c8ad0cbee5bf1d32eb71d1f45796b070880d702fe931e5c62b6b202d04d12e8e"}
Nov 02 07:41:13.216670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.216628    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" event={"ID":"5f869533-f3ef-4491-8275-517bb9d247d7","Type":"ContainerStarted","Data":"d35cad30ce8b54e1a9abc2c910d451dc75b2a12732f15bf548ea229e0fa9bb34"}
Nov 02 07:41:13.216867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.216692    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" event={"ID":"5f869533-f3ef-4491-8275-517bb9d247d7","Type":"ContainerStarted","Data":"b693d7901fb06d22bf8fddf1d24bfd35fd55ac199b5a88a3656e0a00d9a586ff"}
Nov 02 07:41:13.217495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.217470    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:13.220550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.220506    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" event={"ID":"e7da7301-9856-46dc-93de-58745aab4e7c","Type":"ContainerStarted","Data":"0558fe38e9a875fa00f0c03e9686b82e223595cf7786e3a3c2edb88ad17f3b67"}
Nov 02 07:41:13.222021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.221997    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:13.222181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.222164    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" event={"ID":"e7da7301-9856-46dc-93de-58745aab4e7c","Type":"ContainerStarted","Data":"ed9339ecc803d1dc1ea0faf2273bc89c21c6c76b578966d3e4039e5786f7f4f1"}
Nov 02 07:41:13.222301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.222284    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:13.223377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.223356    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:41:13.262359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.262265    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podStartSLOduration=1.262247639 podStartE2EDuration="1.262247639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:13.259990549 +0000 UTC m=+1018.022427459" watchObservedRunningTime="2026-11-02 07:41:13.262247639 +0000 UTC m=+1018.024684544"
Nov 02 07:41:13.302011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.301944    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" podStartSLOduration=2.301926707 podStartE2EDuration="2.301926707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:13.300413731 +0000 UTC m=+1018.062850638" watchObservedRunningTime="2026-11-02 07:41:13.301926707 +0000 UTC m=+1018.064363610"
Nov 02 07:41:13.396592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.396538    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" podStartSLOduration=2.3965247339999998 podStartE2EDuration="2.396524734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:13.393512002 +0000 UTC m=+1018.155948908" watchObservedRunningTime="2026-11-02 07:41:13.396524734 +0000 UTC m=+1018.158961641"
Nov 02 07:41:13.606169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.606140    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:13.615945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.615905    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:13.616209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:13.616178    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:14.591991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.591958    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:14.592382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.592012    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:14.603293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.603258    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:41:14.607963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.607934    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.672097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.672059    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:41:14.715276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715239    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715338    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715484    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715598    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715700    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715859    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715925    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjg5\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.715991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.715958    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.838335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.817184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.838689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.838650    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.845562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.845481    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.845864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.845835    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846084    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846239    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846593    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846732    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846814    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846890    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.846988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.846969    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjg5\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.847045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.847022    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.847642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.847578    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.847763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.847725    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.848099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.848078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.848935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.848911    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.871046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.871014    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjg5\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.871352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.871330    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:14.960660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:14.960623    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:15.315960 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:15.315923    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800e102a_4dd1_46df_bfa3_27a60baa8018.slice/crio-dae3a9700ebba7494c85842188ffafc4d1b431b5d12c09335074c1a8f1b9002d WatchSource:0}: Error finding container dae3a9700ebba7494c85842188ffafc4d1b431b5d12c09335074c1a8f1b9002d: Status 404 returned error can't find the container with id dae3a9700ebba7494c85842188ffafc4d1b431b5d12c09335074c1a8f1b9002d
Nov 02 07:41:15.339940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.324927    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:15.339940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.325029    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:15.339940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.325073    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:15.339940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.336352    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:41:15.591894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.591859    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:15.592083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:15.591916    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:16.297194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:16.297141    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" event={"ID":"800e102a-4dd1-46df-bfa3-27a60baa8018","Type":"ContainerStarted","Data":"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"}
Nov 02 07:41:16.297412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:16.297216    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" event={"ID":"800e102a-4dd1-46df-bfa3-27a60baa8018","Type":"ContainerStarted","Data":"dae3a9700ebba7494c85842188ffafc4d1b431b5d12c09335074c1a8f1b9002d"}
Nov 02 07:41:16.397641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:16.397590    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podStartSLOduration=2.397574748 podStartE2EDuration="2.397574748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:16.381790735 +0000 UTC m=+1021.144227644" watchObservedRunningTime="2026-11-02 07:41:16.397574748 +0000 UTC m=+1021.160011654"
Nov 02 07:41:16.592738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:16.592651    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:16.592738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:16.592707    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:17.062812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.062777    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:17.098357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.078454    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:17.098357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.078529    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:17.618329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.601414    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:17.618329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.601479    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:17.801832 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:41:17.801775    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:41:17.965417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.965250    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:17.965417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:17.965324    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:18.592174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:18.592142    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:18.592392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:18.592197    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:18.972387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:18.971688    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:18.972387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:18.971750    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:19.592058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:19.592023    4515 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:41:19.592238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:19.592085    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 07:41:19.961024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:19.960940    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:19.961024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:19.960997    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:20.961971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:20.961935    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:20.962373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:20.961995    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:20.992400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:20.992362    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:20.992701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:20.992672    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:20.993565 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:20.993546    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:41:21.968334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:21.968061    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:21.968334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:21.968122    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:22.963737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:22.963699    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:22.964006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:22.963973    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:23.659346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.652425    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:23.697895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.697063    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:23.759273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.759136    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:23.862465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.862411    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:23.972802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.972706    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 07:41:23.973098 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:23.973067    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 07:41:24.276342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.275529    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:41:24.597394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.592577    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:41:24.665524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.656586    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:41:24.665524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.657301    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:41:24.711377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.710537    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:41:24.716333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.713035    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:41:24.750490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.747679    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.976394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.969962    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.991583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:24.991536    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:41:25.384474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.360121    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:41:25.601357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.601286    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.602511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.602463    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 02 07:41:25.677397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.662672    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.718127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.717644    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.749351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.749287    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.752322 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.752283    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:41:25.752601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.752586    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:41:25.981384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:25.981264    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.023257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.023210    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:26.023460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.023288    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.048894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.048847    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 07:41:26.522334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.516143    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:26.536421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.533542    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:26.586490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.586456    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:41:26.610747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.610693    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.697134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.697088    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.750044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.750006    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.758343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.758275    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.802394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.801842    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:41:26.815982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.814970    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:41:26.982550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:26.982509    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:27.572221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:27.558844    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:27.600640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:27.600437    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:27.600640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:27.600501    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:27.616342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:27.616276    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:41:27.768454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:27.768402    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.028563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:28.028518    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:41:28.665333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:28.663745    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.686322 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:28.686259    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-99lqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:28.686603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:28.686560    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.801806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:28.801767    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:41:29.032475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:29.030768    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.073508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:29.072458    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:41:29.104007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:29.103067    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:41:46.576334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:46.576267    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:41:47.832622 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:41:47.832588    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:41:51.094403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.094370    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:41:51.102262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.102239    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.186333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.184640    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cc2bcd-openshift-default-dockercfg-g5zgb\""
Nov 02 07:41:51.190056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.190027    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.477515    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.477588    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.477650    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.477696    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.493726    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.493947    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.493994    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.494175    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.527385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.494244    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4855\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615265    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615350    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4855\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615461    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615489    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615532    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615577    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615673    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615732    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.615756    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.626382    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.627047    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.627640    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.654876    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.685362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.655189    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.748373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.722190    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.810979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.810902    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.811413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.811382    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4855\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:51.839955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:51.839921    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\") pod \"e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") " pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:52.083789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.083692    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 07:41:52.090390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.090360    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 07:41:52.090564 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.090457    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:52.090564 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.090468    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181368    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181451    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181516    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181732    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wq8\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.181907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181862    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.182169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181927    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.182169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181951    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.182169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.181977    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.283449    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.283497    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287428    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287465    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287486    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287579    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287665    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287730    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.287840    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wq8\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.289128    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.289470    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.289712    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.290898    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.291185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.291133    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.293869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.293812    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.294632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.294570    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.373983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.373867    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.374260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.374230    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wq8\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.408215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.407987    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:52.579281 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.579247    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:41:52.582864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.582821    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.582973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.582909    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.582973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.582950    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.685024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.684988    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerStarted","Data":"942e47e7fb11ed89a94a578ed230d1b1b8ee0ad3cf844260e47242bf628b1e37"}
Nov 02 07:41:52.948958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.948916    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.949225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.949205    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.949384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.949367    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.986900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:52.980286    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 07:41:53.704155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:53.704117    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerStarted","Data":"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"}
Nov 02 07:41:53.706048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:53.706023    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" event={"ID":"756d4e3e-a850-49eb-b8ff-0bde2e052020","Type":"ContainerStarted","Data":"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"}
Nov 02 07:41:53.706169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:53.706066    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" event={"ID":"756d4e3e-a850-49eb-b8ff-0bde2e052020","Type":"ContainerStarted","Data":"d934ef5a9dd2bce7c3d4752cf856b0280b613ec3a5e37d690d8f4a8a16b8878d"}
Nov 02 07:41:53.778611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:53.778544    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podStartSLOduration=2.778530801 podStartE2EDuration="2.778530801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:53.776398572 +0000 UTC m=+1058.538835477" watchObservedRunningTime="2026-11-02 07:41:53.778530801 +0000 UTC m=+1058.540967706"
Nov 02 07:41:53.850334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:53.850066    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podStartSLOduration=1.850043861 podStartE2EDuration="1.850043861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:53.830466587 +0000 UTC m=+1058.592903498" watchObservedRunningTime="2026-11-02 07:41:53.850043861 +0000 UTC m=+1058.612480769"
Nov 02 07:41:54.102438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.102402    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:54.109892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.109458    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:54.415842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.415759    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:41:54.416174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.416153    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:54.416396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.416367    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:54.761391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.757790    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:54.799279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:54.798976    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:41:55.408477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:55.408443    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:55.408674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:55.408544    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:56.409292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:56.409253    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:56.409787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:56.409348    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:57.409400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:57.408662    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:57.409400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:57.408836    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:58.472343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.414914    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:58.472343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.414993    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:58.536374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.509522    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:41:58.642331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.636351    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:41:58.642331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.636481    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.668647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.667569    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc2bcd\""
Nov 02 07:41:58.681129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.681104    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cc2bcd-dockercfg-xbrhf\""
Nov 02 07:41:58.739843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.739463    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db99z\" (UniqueName: \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.739843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.739524    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.845766 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.845720    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db99z\" (UniqueName: \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.845971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.845787    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.846774 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.846740    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.870623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.870592    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db99z\" (UniqueName: \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\") pod \"payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:58.983135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:58.983088    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:59.287518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.287424    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:59.287518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.287494    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:59.346692 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.346657    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:41:59.348790 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:41:59.348761    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5558072d_db71_433e_80a8_a178a6fcaab0.slice/crio-6505b1134b38a3937cf78f722289bda60cb6cf16b36b6997879ad323dd609f28 WatchSource:0}: Error finding container 6505b1134b38a3937cf78f722289bda60cb6cf16b36b6997879ad323dd609f28: Status 404 returned error can't find the container with id 6505b1134b38a3937cf78f722289bda60cb6cf16b36b6997879ad323dd609f28
Nov 02 07:41:59.409214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.409172    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:41:59.409374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.409243    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:41:59.792382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.792338    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" event={"ID":"5558072d-db71-433e-80a8-a178a6fcaab0","Type":"ContainerStarted","Data":"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"}
Nov 02 07:41:59.792903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.792422    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" event={"ID":"5558072d-db71-433e-80a8-a178a6fcaab0","Type":"ContainerStarted","Data":"6505b1134b38a3937cf78f722289bda60cb6cf16b36b6997879ad323dd609f28"}
Nov 02 07:41:59.792903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.792552    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:41:59.843437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:41:59.843374    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" podStartSLOduration=1.843352758 podStartE2EDuration="1.843352758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:59.836750554 +0000 UTC m=+1064.599187460" watchObservedRunningTime="2026-11-02 07:41:59.843352758 +0000 UTC m=+1064.605789674"
Nov 02 07:42:00.409472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.409425    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 07:42:00.409780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.409740    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 07:42:00.802509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.802476    4515 generic.go:356] "Generic (PLEG): container finished" podID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerID="c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629" exitCode=0
Nov 02 07:42:00.803088 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.802577    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerDied","Data":"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"}
Nov 02 07:42:00.803248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.803226    4515 scope.go:122] "RemoveContainer" containerID="c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"
Nov 02 07:42:00.805238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.805197    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:00.805378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.805293    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:00.805378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.805368    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:00.806288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.806259    4515 generic.go:356] "Generic (PLEG): container finished" podID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerID="74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6" exitCode=0
Nov 02 07:42:00.806401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.806373    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerDied","Data":"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"}
Nov 02 07:42:00.806991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.806972    4515 scope.go:122] "RemoveContainer" containerID="74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"
Nov 02 07:42:00.814982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.813939    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:00.814982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.814019    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:00.814982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:00.814062    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:01.804365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:01.804212    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:42:01.809717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:01.809686    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:42:01.824755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:01.824722    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 07:42:01.959911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:01.931258    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerStarted","Data":"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"}
Nov 02 07:42:01.995048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:01.978625    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerStarted","Data":"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"}
Nov 02 07:42:02.062428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.051894    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:42:02.062428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.052282    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:02.062428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.052495    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:02.663333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.662725    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:42:02.663333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.662876    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:02.663333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:02.662924    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:03.038335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:03.034760    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:03.038335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:03.034824    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:03.664173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:03.663718    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:03.664173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:03.663779    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:04.061344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:04.057575    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:04.061344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:04.057638    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:04.664334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:04.663629    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:04.664334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:04.663693    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:05.045342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:05.045020    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:05.045342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:05.045082    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:05.661332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:05.660911    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:05.661332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:05.660997    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:06.046746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:06.046707    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:06.047015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:06.046988    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:06.668336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:06.667409    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:06.668336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:06.667468    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:07.036771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.036731    4515 patch_prober.go:29] interesting pod/e2e-api-a-d47676-openshift-default-6df488c785-gm487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 07:42:07.048467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.048424    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 07:42:07.240917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.137653    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:42:07.395379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.388499    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.430338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.419482    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b3e5f7\""
Nov 02 07:42:07.472332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.469745    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b3e5f7-dockercfg-grs7p\""
Nov 02 07:42:07.472332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.472182    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.472332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.472224    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzgq\" (UniqueName: \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.581334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.580324    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:42:07.581521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.581477    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.581603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.581521    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzgq\" (UniqueName: \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.583125 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.583066    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.607361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.607333    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzgq\" (UniqueName: \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\") pod \"payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:07.663153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.662896    4515 patch_prober.go:29] interesting pod/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 07:42:07.663153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.662958    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 07:42:07.884344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:07.884284    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:08.041481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.039894    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:42:08.041645 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.041502    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:42:08.045066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.045017    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:42:08.231756 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:42:08.231729    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a477c4c_ff3c_4be7_8bd6_8a50c5798bcd.slice/crio-e7cf72a49224a0ca7df95461582955c639a3ca9b2791536ea7c0b4c4c3ec02cb WatchSource:0}: Error finding container e7cf72a49224a0ca7df95461582955c639a3ca9b2791536ea7c0b4c4c3ec02cb: Status 404 returned error can't find the container with id e7cf72a49224a0ca7df95461582955c639a3ca9b2791536ea7c0b4c4c3ec02cb
Nov 02 07:42:08.233099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.233026    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:42:08.256127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.256054    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" event={"ID":"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd","Type":"ContainerStarted","Data":"e7cf72a49224a0ca7df95461582955c639a3ca9b2791536ea7c0b4c4c3ec02cb"}
Nov 02 07:42:08.666299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.666267    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:42:08.666839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.666813    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:42:08.667979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:08.667957    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:42:09.263625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:09.263581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" event={"ID":"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd","Type":"ContainerStarted","Data":"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"}
Nov 02 07:42:09.264038 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:09.263657    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:09.305562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:09.305513    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" podStartSLOduration=2.305499451 podStartE2EDuration="2.305499451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:09.304911651 +0000 UTC m=+1074.067348556" watchObservedRunningTime="2026-11-02 07:42:09.305499451 +0000 UTC m=+1074.067936356"
Nov 02 07:42:10.810155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:10.810127    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:42:11.062621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:11.062540    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:42:17.766679 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:42:17.764886    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:42:20.273456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:20.273426    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:42:28.031766 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:28.031734    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"]
Nov 02 07:42:28.043906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:28.043872    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894bb881-f5lfb"]
Nov 02 07:42:29.374295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:29.374258    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0799b7b9-5e28-45f2-ac0d-e6b3bda50762" path="/var/lib/kubelet/pods/0799b7b9-5e28-45f2-ac0d-e6b3bda50762/volumes"
Nov 02 07:42:44.502456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:44.502409    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:42:44.544554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:44.544524    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:42:46.711332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.705459    4515 generic.go:356] "Generic (PLEG): container finished" podID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerID="8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e" exitCode=0
Nov 02 07:42:46.711332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.705527    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerDied","Data":"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"}
Nov 02 07:42:46.711332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.706094    4515 scope.go:122] "RemoveContainer" containerID="8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"
Nov 02 07:42:46.723966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.722521    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:46.724637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.724264    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:46.724637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:46.724375    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:47.724957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:47.723787    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerStarted","Data":"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"}
Nov 02 07:42:47.752373 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:42:47.752336    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:42:48.091116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:48.091082    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:42:48.091382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:48.091342    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:42:48.091471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:48.091397    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 07:42:49.091160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.091121    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:42:49.091652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.091188    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 07:42:49.503726 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.503657    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" podUID="e7da7301-9856-46dc-93de-58745aab4e7c" containerName="payload-processing" containerID="cri-o://0558fe38e9a875fa00f0c03e9686b82e223595cf7786e3a3c2edb88ad17f3b67" gracePeriod=25
Nov 02 07:42:49.545721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.545674    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" podUID="5f869533-f3ef-4491-8275-517bb9d247d7" containerName="payload-pre-processing" containerID="cri-o://d35cad30ce8b54e1a9abc2c910d451dc75b2a12732f15bf548ea229e0fa9bb34" gracePeriod=25
Nov 02 07:42:49.759989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.758854    4515 generic.go:356] "Generic (PLEG): container finished" podID="5f869533-f3ef-4491-8275-517bb9d247d7" containerID="d35cad30ce8b54e1a9abc2c910d451dc75b2a12732f15bf548ea229e0fa9bb34" exitCode=0
Nov 02 07:42:49.759989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.758969    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" event={"ID":"5f869533-f3ef-4491-8275-517bb9d247d7","Type":"ContainerDied","Data":"d35cad30ce8b54e1a9abc2c910d451dc75b2a12732f15bf548ea229e0fa9bb34"}
Nov 02 07:42:49.762794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.762423    4515 generic.go:356] "Generic (PLEG): container finished" podID="e7da7301-9856-46dc-93de-58745aab4e7c" containerID="0558fe38e9a875fa00f0c03e9686b82e223595cf7786e3a3c2edb88ad17f3b67" exitCode=0
Nov 02 07:42:49.762794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.762603    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" event={"ID":"e7da7301-9856-46dc-93de-58745aab4e7c","Type":"ContainerDied","Data":"0558fe38e9a875fa00f0c03e9686b82e223595cf7786e3a3c2edb88ad17f3b67"}
Nov 02 07:42:49.890938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.890911    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:42:49.955501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.955474    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:42:49.985138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.985097    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\" (UniqueName: \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\") pod \"5f869533-f3ef-4491-8275-517bb9d247d7\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") "
Nov 02 07:42:49.985425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.985401    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\") pod \"5f869533-f3ef-4491-8275-517bb9d247d7\" (UID: \"5f869533-f3ef-4491-8275-517bb9d247d7\") "
Nov 02 07:42:49.985539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.985477    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\") pod \"e7da7301-9856-46dc-93de-58745aab4e7c\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") "
Nov 02 07:42:49.985617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.985602    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\" (UniqueName: \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\") pod \"e7da7301-9856-46dc-93de-58745aab4e7c\" (UID: \"e7da7301-9856-46dc-93de-58745aab4e7c\") "
Nov 02 07:42:49.985955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.985915    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume" pod "5f869533-f3ef-4491-8275-517bb9d247d7" (UID: "5f869533-f3ef-4491-8275-517bb9d247d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:42:49.986505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.986233    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f869533-f3ef-4491-8275-517bb9d247d7-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:42:49.986505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.986422    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume" pod "e7da7301-9856-46dc-93de-58745aab4e7c" (UID: "e7da7301-9856-46dc-93de-58745aab4e7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:42:49.987967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.987925    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj" pod "5f869533-f3ef-4491-8275-517bb9d247d7" (UID: "5f869533-f3ef-4491-8275-517bb9d247d7"). InnerVolumeSpecName "kube-api-access-nq7zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:49.988407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:49.988382    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp" pod "e7da7301-9856-46dc-93de-58745aab4e7c" (UID: "e7da7301-9856-46dc-93de-58745aab4e7c"). InnerVolumeSpecName "kube-api-access-sgbxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:50.088411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.088299    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq7zj\" (UniqueName: \"kubernetes.io/projected/5f869533-f3ef-4491-8275-517bb9d247d7-kube-api-access-nq7zj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:42:50.088411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.088361    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7da7301-9856-46dc-93de-58745aab4e7c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:42:50.088411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.088377    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbxp\" (UniqueName: \"kubernetes.io/projected/e7da7301-9856-46dc-93de-58745aab4e7c-kube-api-access-sgbxp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:42:50.091532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.091505    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:42:50.091877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.091555    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 07:42:50.348671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.348581    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-8fa938"]
Nov 02 07:42:50.349279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349261    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e7da7301-9856-46dc-93de-58745aab4e7c" containerName="payload-processing"
Nov 02 07:42:50.349355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349279    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7da7301-9856-46dc-93de-58745aab4e7c" containerName="payload-processing"
Nov 02 07:42:50.349355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349291    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5f869533-f3ef-4491-8275-517bb9d247d7" containerName="payload-pre-processing"
Nov 02 07:42:50.349355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349297    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f869533-f3ef-4491-8275-517bb9d247d7" containerName="payload-pre-processing"
Nov 02 07:42:50.349452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349447    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7da7301-9856-46dc-93de-58745aab4e7c" containerName="payload-processing"
Nov 02 07:42:50.349482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.349455    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f869533-f3ef-4491-8275-517bb9d247d7" containerName="payload-pre-processing"
Nov 02 07:42:50.353118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.353100    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:50.368105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.368069    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-8fa938"]
Nov 02 07:42:50.368997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.368966    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmvw6\""
Nov 02 07:42:50.445333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.441713    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fz5\" (UniqueName: \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\") pod \"test-curl-7355-8fa938\" (UID: \"d54faa98-9577-4271-9eae-635445c5ffac\") " pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:50.567345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.561645    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fz5\" (UniqueName: \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\") pod \"test-curl-7355-8fa938\" (UID: \"d54faa98-9577-4271-9eae-635445c5ffac\") " pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:50.669515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.669417    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fz5\" (UniqueName: \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\") pod \"test-curl-7355-8fa938\" (UID: \"d54faa98-9577-4271-9eae-635445c5ffac\") " pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:50.772686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.772655    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"
Nov 02 07:42:50.772854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.772658    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p" event={"ID":"5f869533-f3ef-4491-8275-517bb9d247d7","Type":"ContainerDied","Data":"b693d7901fb06d22bf8fddf1d24bfd35fd55ac199b5a88a3656e0a00d9a586ff"}
Nov 02 07:42:50.772854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.772795    4515 scope.go:122] "RemoveContainer" containerID="d35cad30ce8b54e1a9abc2c910d451dc75b2a12732f15bf548ea229e0fa9bb34"
Nov 02 07:42:50.774107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.774074    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"
Nov 02 07:42:50.774176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.774114    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4" event={"ID":"e7da7301-9856-46dc-93de-58745aab4e7c","Type":"ContainerDied","Data":"ed9339ecc803d1dc1ea0faf2273bc89c21c6c76b578966d3e4039e5786f7f4f1"}
Nov 02 07:42:50.783538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.783520    4515 scope.go:122] "RemoveContainer" containerID="0558fe38e9a875fa00f0c03e9686b82e223595cf7786e3a3c2edb88ad17f3b67"
Nov 02 07:42:50.846190 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.846149    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:42:50.849973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.849950    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d47676-76ff6c4867-svq2p"]
Nov 02 07:42:50.881634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.881597    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:42:50.887956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.887927    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d47676-668589f87f-kz8h4"]
Nov 02 07:42:50.966923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:50.966846    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:51.091515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.091478    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:42:51.091676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.091548    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 07:42:51.297802 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:42:51.297755    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd54faa98_9577_4271_9eae_635445c5ffac.slice/crio-554bf8d0d7e11bb37aee0c2da84bc313adeb533af87d501935639126ac2d9714 WatchSource:0}: Error finding container 554bf8d0d7e11bb37aee0c2da84bc313adeb533af87d501935639126ac2d9714: Status 404 returned error can't find the container with id 554bf8d0d7e11bb37aee0c2da84bc313adeb533af87d501935639126ac2d9714
Nov 02 07:42:51.298737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.298715    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-8fa938"]
Nov 02 07:42:51.374091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.374055    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f869533-f3ef-4491-8275-517bb9d247d7" path="/var/lib/kubelet/pods/5f869533-f3ef-4491-8275-517bb9d247d7/volumes"
Nov 02 07:42:51.374593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.374577    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7da7301-9856-46dc-93de-58745aab4e7c" path="/var/lib/kubelet/pods/e7da7301-9856-46dc-93de-58745aab4e7c/volumes"
Nov 02 07:42:51.783250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:51.783211    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-8fa938" event={"ID":"d54faa98-9577-4271-9eae-635445c5ffac","Type":"ContainerStarted","Data":"554bf8d0d7e11bb37aee0c2da84bc313adeb533af87d501935639126ac2d9714"}
Nov 02 07:42:52.091810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:52.091718    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:42:52.091810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:52.091777    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 07:42:53.096266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:53.096233    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:42:53.096762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:53.096690    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:42:53.097477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:53.097455    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:42:53.800681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:53.800642    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-8fa938" event={"ID":"d54faa98-9577-4271-9eae-635445c5ffac","Type":"ContainerStarted","Data":"3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c"}
Nov 02 07:42:53.839840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:53.839786    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7355-8fa938" podStartSLOduration=1.47417881 podStartE2EDuration="3.839772582s" totalImagesPullingTime="2.365593772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:53.839506727 +0000 UTC m=+1118.601943630" watchObservedRunningTime="2026-11-02 07:42:53.839772582 +0000 UTC m=+1118.602209488"
Nov 02 07:42:54.723437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:54.723390    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-8fa938"]
Nov 02 07:42:54.727616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:54.727590    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-8fa938"]
Nov 02 07:42:55.813883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:55.813847    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7355-8fa938" podUID="d54faa98-9577-4271-9eae-635445c5ffac" containerName="test-curl-7355-8fa938" containerID="cri-o://3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c" gracePeriod=2
Nov 02 07:42:58.136658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.136632    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:58.234143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.234108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\" (UniqueName: \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\") pod \"d54faa98-9577-4271-9eae-635445c5ffac\" (UID: \"d54faa98-9577-4271-9eae-635445c5ffac\") "
Nov 02 07:42:58.236152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.236127    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5" pod "d54faa98-9577-4271-9eae-635445c5ffac" (UID: "d54faa98-9577-4271-9eae-635445c5ffac"). InnerVolumeSpecName "kube-api-access-42fz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:58.336629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.336593    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fz5\" (UniqueName: \"kubernetes.io/projected/d54faa98-9577-4271-9eae-635445c5ffac-kube-api-access-42fz5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:42:58.834973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.834935    4515 generic.go:356] "Generic (PLEG): container finished" podID="d54faa98-9577-4271-9eae-635445c5ffac" containerID="3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c" exitCode=137
Nov 02 07:42:58.835156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.834994    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:58.835156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.835039    4515 scope.go:122] "RemoveContainer" containerID="3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c"
Nov 02 07:42:58.845849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.845830    4515 scope.go:122] "RemoveContainer" containerID="3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c"
Nov 02 07:42:58.846101 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:42:58.846081    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c\": container with ID starting with 3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c not found: ID does not exist" containerID="3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c"
Nov 02 07:42:58.846159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:58.846115    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c"} err="failed to get container status \"3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c\": rpc error: code = NotFound desc = could not find container \"3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c\": container with ID starting with 3059abcd34910dbefd985ef2e05b6e334c99f0f733f4705cc9648d51e7caf75c not found: ID does not exist"
Nov 02 07:42:58.859833 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:42:58.859802    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7355-8fa938\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="d54faa98-9577-4271-9eae-635445c5ffac" pod="opendatahub/test-curl-7355-8fa938"
Nov 02 07:42:59.375085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:42:59.375050    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54faa98-9577-4271-9eae-635445c5ffac" path="/var/lib/kubelet/pods/d54faa98-9577-4271-9eae-635445c5ffac/volumes"
Nov 02 07:43:01.049472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:01.049436    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:43:05.973119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:05.973086    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:43:06.019436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:06.019404    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:43:07.292619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.292574    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:43:07.296632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.294101    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d54faa98-9577-4271-9eae-635445c5ffac" containerName="test-curl-7355-8fa938"
Nov 02 07:43:07.296857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.296835    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54faa98-9577-4271-9eae-635445c5ffac" containerName="test-curl-7355-8fa938"
Nov 02 07:43:07.297255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.297232    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54faa98-9577-4271-9eae-635445c5ffac" containerName="test-curl-7355-8fa938"
Nov 02 07:43:07.301657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.301612    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.322426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.316600    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:43:07.398385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398351    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398421    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398456    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398562    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398647    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398704    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.398898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.398858    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvr9r\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.399104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.399088    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.399248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.399234    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500173    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvr9r\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500237    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500295    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500361    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500400    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500452    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500501    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500562    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500629    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500735    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500761    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500838    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.500998 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.500960    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.501138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.501118    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.502902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.502873    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.503064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.503047    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.524966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.524939    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvr9r\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.525259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.525239    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.666993 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.666903    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:07.995477 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:07.995447    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaad616_829f_48e7_b900_46b36aa1922d.slice/crio-01d3b8f537e5d9e64a1938f75a5f703f13577ff4e18d080c03fa0e76501edf0e WatchSource:0}: Error finding container 01d3b8f537e5d9e64a1938f75a5f703f13577ff4e18d080c03fa0e76501edf0e: Status 404 returned error can't find the container with id 01d3b8f537e5d9e64a1938f75a5f703f13577ff4e18d080c03fa0e76501edf0e
Nov 02 07:43:07.997288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.997267    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:43:07.998016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.997983    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:07.998069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.998051    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:07.998104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:07.998080    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:08.958369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:08.958332    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" event={"ID":"0eaad616-829f-48e7-b900-46b36aa1922d","Type":"ContainerStarted","Data":"3439eacb87b4f5f0685986c914e1ea5b986faea4d2cfc2c7025625629e46fc88"}
Nov 02 07:43:08.958775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:08.958387    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" event={"ID":"0eaad616-829f-48e7-b900-46b36aa1922d","Type":"ContainerStarted","Data":"01d3b8f537e5d9e64a1938f75a5f703f13577ff4e18d080c03fa0e76501edf0e"}
Nov 02 07:43:09.046937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:09.046883    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podStartSLOduration=2.046870155 podStartE2EDuration="2.046870155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:09.033826992 +0000 UTC m=+1133.796263898" watchObservedRunningTime="2026-11-02 07:43:09.046870155 +0000 UTC m=+1133.809307060"
Nov 02 07:43:09.667744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:09.667709    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:09.669191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:09.669155    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:09.669343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:09.669209    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:10.452505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:10.448673    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:43:10.452505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:10.448949    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy" containerID="cri-o://744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710" gracePeriod=30
Nov 02 07:43:10.684732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:10.684626    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:10.684732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:10.684686    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:10.974085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:10.974045    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing" containerID="cri-o://38a8b0de284adbf5820cd75a5c348f8da34d1e381ba65c9c65ab4a482c3e1d4f" gracePeriod=25
Nov 02 07:43:11.039636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:11.039404    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing" containerID="cri-o://7f79c00916bb7f00f7dedf5b1658ce4350ef8acb9f57ef634366e7c65c842c4b" gracePeriod=25
Nov 02 07:43:11.668078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:11.668042    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:11.668586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:11.668116    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:11.698922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:11.698890    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:43:11.699076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:11.698889    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:43:12.667953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:12.667920    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:12.668146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:12.667987    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:13.725335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:13.723012    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:13.725335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:13.723072    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:14.667964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:14.667926    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 07:43:14.668148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:14.667998    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 07:43:15.232264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.228486    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:43:15.239123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.239097    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.260890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.260851    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:43:15.262243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262200    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262270    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262359    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262405    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4np2t\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262494    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262528    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262550    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262582    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.262693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.262651    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.267244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.267220    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7e07c6-openshift-default-dockercfg-wnx8l\""
Nov 02 07:43:15.459403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.413748    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501580    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501629    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501692    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4np2t\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501759    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501811    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501844    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501896    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.503406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.501990    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.529333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.519825    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.529333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.528970    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.529692 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.529359    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.533335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.531095    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.533335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.531440    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.662725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.661214    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.662725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.661717    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.663818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.663792    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.700333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.699264    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4np2t\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\") pod \"e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") " pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.922125 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.922089    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7e07c6-openshift-default-dockercfg-wnx8l\""
Nov 02 07:43:15.933449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.933415    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:15.943999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:15.942963    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.037793    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.037912    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038059    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038144    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038188    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038386    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038441    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038468    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.038500    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\") pod \"fb993855-61ba-4214-9eb6-1d958e1e37d8\" (UID: \"fb993855-61ba-4214-9eb6-1d958e1e37d8\") "
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.039929    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.040188    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.040496    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.042347    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:16.044495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.042353    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051197    4515 generic.go:356] "Generic (PLEG): container finished" podID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerID="744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710" exitCode=0
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051271    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerDied","Data":"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"}
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051338    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487" event={"ID":"fb993855-61ba-4214-9eb6-1d958e1e37d8","Type":"ContainerDied","Data":"9b7eb745666a197639fd4d9eddc061db3a77e0c8b5d17c3520ae3c7daea3730e"}
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051373    4515 scope.go:122] "RemoveContainer" containerID="744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051527    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"
Nov 02 07:43:16.051926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.051817    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "kube-api-access-jdl2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:16.053377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.052680    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:43:16.053377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.053175    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:16.053377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.053277    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy" pod "fb993855-61ba-4214-9eb6-1d958e1e37d8" (UID: "fb993855-61ba-4214-9eb6-1d958e1e37d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:16.067426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.067201    4515 scope.go:122] "RemoveContainer" containerID="c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"
Nov 02 07:43:16.082158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.082086    4515 scope.go:122] "RemoveContainer" containerID="744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"
Nov 02 07:43:16.082605 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:16.082574    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710\": container with ID starting with 744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710 not found: ID does not exist" containerID="744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"
Nov 02 07:43:16.082769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.082619    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710"} err="failed to get container status \"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710\": rpc error: code = NotFound desc = could not find container \"744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710\": container with ID starting with 744f427ff2b6c22a30209b953224c843bdf7638192a82ca34e1cfad525c71710 not found: ID does not exist"
Nov 02 07:43:16.082769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.082644    4515 scope.go:122] "RemoveContainer" containerID="c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"
Nov 02 07:43:16.082995 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:16.082967    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629\": container with ID starting with c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629 not found: ID does not exist" containerID="c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"
Nov 02 07:43:16.083064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.082998    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629"} err="failed to get container status \"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629\": rpc error: code = NotFound desc = could not find container \"c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629\": container with ID starting with c1c441e44381dd7ef7dbddb71cfa9e568211aad6cf20190e80735acf9becf629 not found: ID does not exist"
Nov 02 07:43:16.140207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140166    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140201    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140215    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140226    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140239    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140249    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb993855-61ba-4214-9eb6-1d958e1e37d8-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140260    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb993855-61ba-4214-9eb6-1d958e1e37d8-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140272    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdl2d\" (UniqueName: \"kubernetes.io/projected/fb993855-61ba-4214-9eb6-1d958e1e37d8-kube-api-access-jdl2d\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.140492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.140283    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb993855-61ba-4214-9eb6-1d958e1e37d8-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:16.278751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.278723    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:16.279172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.278784    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:43:16.279620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.279561    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:16.280938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.280911    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:43:16.289480 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:16.283383    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962061a8_44c3_4e04_8467_9eeab4761cea.slice/crio-271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73 WatchSource:0}: Error finding container 271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73: Status 404 returned error can't find the container with id 271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73
Nov 02 07:43:16.289480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.286409    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:16.289480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.286666    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:16.289480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.286720    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:16.447386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.446385    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:43:16.449909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:16.449828    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d47676-openshift-default-6df488c785-gm487"]
Nov 02 07:43:17.061708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.061672    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerStarted","Data":"50add28e2b3afcad546bbe170590b1bcc8d78be9f1b99f9f19bae6f4a9741538"}
Nov 02 07:43:17.061895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.061728    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerStarted","Data":"271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73"}
Nov 02 07:43:17.116844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.116787    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podStartSLOduration=2.116772921 podStartE2EDuration="2.116772921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:17.113662019 +0000 UTC m=+1141.876098925" watchObservedRunningTime="2026-11-02 07:43:17.116772921 +0000 UTC m=+1141.879209830"
Nov 02 07:43:17.374728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.374650    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" path="/var/lib/kubelet/pods/fb993855-61ba-4214-9eb6-1d958e1e37d8/volumes"
Nov 02 07:43:17.724235 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:17.724144    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:43:17.915107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.915075    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:43:17.934135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.934095    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:17.940016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:17.939993    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:18.069343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:18.068942    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:18.069882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:18.069853    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:43:21.651886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.651805    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:43:21.652473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.652458    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.652522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.652475    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.652522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.652487    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.652522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.652494    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.652627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.652613    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.657445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.657420    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.704329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.702424    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e07c6\""
Nov 02 07:43:21.727810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.727776    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e07c6-dockercfg-qk278\""
Nov 02 07:43:21.750589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.743655    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.750589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.743822    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnqj\" (UniqueName: \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.750589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.745448    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:43:21.770357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.752510    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:43:21.770357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.756402    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:43:21.802332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.798584    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:43:21.818390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.807568    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb993855-61ba-4214-9eb6-1d958e1e37d8" containerName="istio-proxy"
Nov 02 07:43:21.880334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.852156    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.880334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.871721    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnqj\" (UniqueName: \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.880334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.860559    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.928744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.910716    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:21.954347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.950039    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnqj\" (UniqueName: \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:21.996407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.991467    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjmr\" (UniqueName: \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:21.996407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:21.991600    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.014465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.012569    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:22.069679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.069645    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:43:22.097382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.097020    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjmr\" (UniqueName: \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.097382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.097144    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.100846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.098919    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.121806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.121519    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjmr\" (UniqueName: \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\") pod \"payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.349305 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.349265    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:22.395598 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:22.395559    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c0ec82_e6dd_4f7c_82b1_f83d6007533a.slice/crio-77c0d70d6853d37ae35ffd6fb5aac9932ef5b84fe70767b623a157ff8a5ba211 WatchSource:0}: Error finding container 77c0d70d6853d37ae35ffd6fb5aac9932ef5b84fe70767b623a157ff8a5ba211: Status 404 returned error can't find the container with id 77c0d70d6853d37ae35ffd6fb5aac9932ef5b84fe70767b623a157ff8a5ba211
Nov 02 07:43:22.396748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.396717    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:43:22.659245 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:22.659208    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212b6fe9_3ece_460c_9f01_2953f3bc4c8c.slice/crio-36749a8efb12d6e8d11f56106fac40ee4e494acf90c5bb90279dea4b133d870a WatchSource:0}: Error finding container 36749a8efb12d6e8d11f56106fac40ee4e494acf90c5bb90279dea4b133d870a: Status 404 returned error can't find the container with id 36749a8efb12d6e8d11f56106fac40ee4e494acf90c5bb90279dea4b133d870a
Nov 02 07:43:22.660010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.659987    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:43:22.928405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:22.916543    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" podUID="5558072d-db71-433e-80a8-a178a6fcaab0" containerName="payload-processing" containerID="cri-o://bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e" gracePeriod=25
Nov 02 07:43:23.175204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.173578    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" event={"ID":"48c0ec82-e6dd-4f7c-82b1-f83d6007533a","Type":"ContainerStarted","Data":"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"}
Nov 02 07:43:23.175204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.173652    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" event={"ID":"48c0ec82-e6dd-4f7c-82b1-f83d6007533a","Type":"ContainerStarted","Data":"77c0d70d6853d37ae35ffd6fb5aac9932ef5b84fe70767b623a157ff8a5ba211"}
Nov 02 07:43:23.175204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.174368    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:23.177358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.176815    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" event={"ID":"212b6fe9-3ece-460c-9f01-2953f3bc4c8c","Type":"ContainerStarted","Data":"a9ca997fb6090785598028398d1f4c3ab50bc4cf02d024c6ee260784ed1980a0"}
Nov 02 07:43:23.177358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.176861    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" event={"ID":"212b6fe9-3ece-460c-9f01-2953f3bc4c8c","Type":"ContainerStarted","Data":"36749a8efb12d6e8d11f56106fac40ee4e494acf90c5bb90279dea4b133d870a"}
Nov 02 07:43:23.177754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.177720    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:23.244768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.244718    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" podStartSLOduration=2.244703548 podStartE2EDuration="2.244703548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:23.242039682 +0000 UTC m=+1148.004476583" watchObservedRunningTime="2026-11-02 07:43:23.244703548 +0000 UTC m=+1148.007140506"
Nov 02 07:43:23.287947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.287882    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" podStartSLOduration=2.287861517 podStartE2EDuration="2.287861517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:23.286133128 +0000 UTC m=+1148.048570033" watchObservedRunningTime="2026-11-02 07:43:23.287861517 +0000 UTC m=+1148.050298424"
Nov 02 07:43:23.359144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.359118    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:43:23.444664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.444583    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\" (UniqueName: \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\") pod \"5558072d-db71-433e-80a8-a178a6fcaab0\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") "
Nov 02 07:43:23.445021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.444996    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\") pod \"5558072d-db71-433e-80a8-a178a6fcaab0\" (UID: \"5558072d-db71-433e-80a8-a178a6fcaab0\") "
Nov 02 07:43:23.475348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.450735    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume" pod "5558072d-db71-433e-80a8-a178a6fcaab0" (UID: "5558072d-db71-433e-80a8-a178a6fcaab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:23.476171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.476137    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5558072d-db71-433e-80a8-a178a6fcaab0-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:23.511371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.506661    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z" pod "5558072d-db71-433e-80a8-a178a6fcaab0" (UID: "5558072d-db71-433e-80a8-a178a6fcaab0"). InnerVolumeSpecName "kube-api-access-db99z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:23.579031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:23.578992    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db99z\" (UniqueName: \"kubernetes.io/projected/5558072d-db71-433e-80a8-a178a6fcaab0-kube-api-access-db99z\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:24.184215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.184180    4515 generic.go:356] "Generic (PLEG): container finished" podID="5558072d-db71-433e-80a8-a178a6fcaab0" containerID="bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e" exitCode=0
Nov 02 07:43:24.184620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.184231    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"
Nov 02 07:43:24.184620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.184266    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" event={"ID":"5558072d-db71-433e-80a8-a178a6fcaab0","Type":"ContainerDied","Data":"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"}
Nov 02 07:43:24.184620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.184340    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz" event={"ID":"5558072d-db71-433e-80a8-a178a6fcaab0","Type":"ContainerDied","Data":"6505b1134b38a3937cf78f722289bda60cb6cf16b36b6997879ad323dd609f28"}
Nov 02 07:43:24.184620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.184371    4515 scope.go:122] "RemoveContainer" containerID="bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"
Nov 02 07:43:24.195255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.195235    4515 scope.go:122] "RemoveContainer" containerID="bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"
Nov 02 07:43:24.195608 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:24.195588    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e\": container with ID starting with bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e not found: ID does not exist" containerID="bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"
Nov 02 07:43:24.195680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.195621    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e"} err="failed to get container status \"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e\": rpc error: code = NotFound desc = could not find container \"bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e\": container with ID starting with bcf8f518771ebf2bc8f16f339a43aa6d9c6a388795a4f728676b1f1fc024140e not found: ID does not exist"
Nov 02 07:43:24.240944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.240914    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:43:24.253527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:24.253497    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc2bcd-84dfcc7d75-pdvzz"]
Nov 02 07:43:25.422270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:25.422220    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5558072d-db71-433e-80a8-a178a6fcaab0" path="/var/lib/kubelet/pods/5558072d-db71-433e-80a8-a178a6fcaab0/volumes"
Nov 02 07:43:25.653077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:25.653031    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:43:25.653370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:25.653329    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy" containerID="cri-o://156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431" gracePeriod=30
Nov 02 07:43:30.946919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:30.946894    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:43:31.055820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.055792    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.055966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.055835    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.055966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.055868    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.055966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.055898    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056063    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056105    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056175    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056204    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056240    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\") pod \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\" (UID: \"16fc34ba-3dc6-4c8d-9dbb-b62a18064652\") "
Nov 02 07:43:31.056888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056380    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:31.056888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.056596    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:31.057661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.057142    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.057661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.057169    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.057661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.057404    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:31.057661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.057561    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:31.058535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.058507    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:31.059765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.059733    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:43:31.059876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.059780    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:31.059944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.059887    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:31.060552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.060532    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw" pod "16fc34ba-3dc6-4c8d-9dbb-b62a18064652" (UID: "16fc34ba-3dc6-4c8d-9dbb-b62a18064652"). InnerVolumeSpecName "kube-api-access-n8sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:31.159702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159659    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159710    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159725    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159739    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159773    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8sqw\" (UniqueName: \"kubernetes.io/projected/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-kube-api-access-n8sqw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159788    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.159838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.159803    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fc34ba-3dc6-4c8d-9dbb-b62a18064652-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.255294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.255261    4515 generic.go:356] "Generic (PLEG): container finished" podID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerID="38a8b0de284adbf5820cd75a5c348f8da34d1e381ba65c9c65ab4a482c3e1d4f" exitCode=0
Nov 02 07:43:31.255509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.255352    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" event={"ID":"0970f1e5-5b88-44b2-a9dd-8f6266da5a34","Type":"ContainerDied","Data":"38a8b0de284adbf5820cd75a5c348f8da34d1e381ba65c9c65ab4a482c3e1d4f"}
Nov 02 07:43:31.257368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.257340    4515 generic.go:356] "Generic (PLEG): container finished" podID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerID="156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431" exitCode=0
Nov 02 07:43:31.257494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.257403    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"
Nov 02 07:43:31.257494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.257467    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerDied","Data":"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"}
Nov 02 07:43:31.257627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.257507    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx" event={"ID":"16fc34ba-3dc6-4c8d-9dbb-b62a18064652","Type":"ContainerDied","Data":"20c4ac1587c074e28ced08114bdfee7473a7dcbb939710c58d626bdfd8045197"}
Nov 02 07:43:31.257627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.257535    4515 scope.go:122] "RemoveContainer" containerID="156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"
Nov 02 07:43:31.263718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.263690    4515 generic.go:356] "Generic (PLEG): container finished" podID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerID="7f79c00916bb7f00f7dedf5b1658ce4350ef8acb9f57ef634366e7c65c842c4b" exitCode=0
Nov 02 07:43:31.263845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.263764    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" event={"ID":"5254b663-d289-4fc1-abcf-ebd589039ee9","Type":"ContainerDied","Data":"7f79c00916bb7f00f7dedf5b1658ce4350ef8acb9f57ef634366e7c65c842c4b"}
Nov 02 07:43:31.280402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.279849    4515 scope.go:122] "RemoveContainer" containerID="74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"
Nov 02 07:43:31.299272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.299253    4515 scope.go:122] "RemoveContainer" containerID="156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"
Nov 02 07:43:31.299801 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:31.299776    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431\": container with ID starting with 156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431 not found: ID does not exist" containerID="156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"
Nov 02 07:43:31.299918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.299810    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431"} err="failed to get container status \"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431\": rpc error: code = NotFound desc = could not find container \"156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431\": container with ID starting with 156fbab4e495093fb2c65bc57759d0de7ec82273d37e9d456c7cc880066c1431 not found: ID does not exist"
Nov 02 07:43:31.299918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.299829    4515 scope.go:122] "RemoveContainer" containerID="74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"
Nov 02 07:43:31.300143 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:31.300122    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6\": container with ID starting with 74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6 not found: ID does not exist" containerID="74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"
Nov 02 07:43:31.300209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.300152    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6"} err="failed to get container status \"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6\": rpc error: code = NotFound desc = could not find container \"74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6\": container with ID starting with 74723d6cf41e99062bcdea6bd0def831688a052a2f5fd7afbd56df0f313190e6 not found: ID does not exist"
Nov 02 07:43:31.320102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.320075    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:43:31.350462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.350436    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:43:31.353721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.353696    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fe085e-openshift-default-7cfdb79575-pm4bx"]
Nov 02 07:43:31.358134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.358113    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:43:31.373419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.373391    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" path="/var/lib/kubelet/pods/16fc34ba-3dc6-4c8d-9dbb-b62a18064652/volumes"
Nov 02 07:43:31.464182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.463938    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\") pod \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") "
Nov 02 07:43:31.464397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.464350    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume" pod "0970f1e5-5b88-44b2-a9dd-8f6266da5a34" (UID: "0970f1e5-5b88-44b2-a9dd-8f6266da5a34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:31.464591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.464522    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\" (UniqueName: \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\") pod \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\" (UID: \"0970f1e5-5b88-44b2-a9dd-8f6266da5a34\") "
Nov 02 07:43:31.465264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.464831    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\" (UniqueName: \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\") pod \"5254b663-d289-4fc1-abcf-ebd589039ee9\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") "
Nov 02 07:43:31.465893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.465470    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\") pod \"5254b663-d289-4fc1-abcf-ebd589039ee9\" (UID: \"5254b663-d289-4fc1-abcf-ebd589039ee9\") "
Nov 02 07:43:31.465893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.465859    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume" pod "5254b663-d289-4fc1-abcf-ebd589039ee9" (UID: "5254b663-d289-4fc1-abcf-ebd589039ee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:31.466818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.466795    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5254b663-d289-4fc1-abcf-ebd589039ee9-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.466961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.466937    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.467712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.467684    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2" pod "0970f1e5-5b88-44b2-a9dd-8f6266da5a34" (UID: "0970f1e5-5b88-44b2-a9dd-8f6266da5a34"). InnerVolumeSpecName "kube-api-access-bpmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:31.468697 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.468672    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k" pod "5254b663-d289-4fc1-abcf-ebd589039ee9" (UID: "5254b663-d289-4fc1-abcf-ebd589039ee9"). InnerVolumeSpecName "kube-api-access-zw92k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:31.568843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.568811    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw92k\" (UniqueName: \"kubernetes.io/projected/5254b663-d289-4fc1-abcf-ebd589039ee9-kube-api-access-zw92k\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:31.568843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:31.568839    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmq2\" (UniqueName: \"kubernetes.io/projected/0970f1e5-5b88-44b2-a9dd-8f6266da5a34-kube-api-access-bpmq2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:32.039511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.039410    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:43:32.041853 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.040550    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" podUID="9410f552-92c8-41dd-8833-83f9c7a99f1f" containerName="maas-api" containerID="cri-o://78ccfea6b6471d8735c86673acfd98ea46fe2bb69dbb0ce9fd79ff15fb029e2e" gracePeriod=30
Nov 02 07:43:32.273410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.273377    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"
Nov 02 07:43:32.273807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.273380    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2" event={"ID":"5254b663-d289-4fc1-abcf-ebd589039ee9","Type":"ContainerDied","Data":"2327b010a71cb986db37a1e65b006b902af41ef51097f5f089af8a9f91c177dc"}
Nov 02 07:43:32.273807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.273607    4515 scope.go:122] "RemoveContainer" containerID="7f79c00916bb7f00f7dedf5b1658ce4350ef8acb9f57ef634366e7c65c842c4b"
Nov 02 07:43:32.275879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.275847    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk" event={"ID":"0970f1e5-5b88-44b2-a9dd-8f6266da5a34","Type":"ContainerDied","Data":"75a0c156ec1cb979a25f733f3842c08c9a26e073155b4a4fb8f9e36112c03bb2"}
Nov 02 07:43:32.276000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.275950    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"
Nov 02 07:43:32.488332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.485758    4515 generic.go:356] "Generic (PLEG): container finished" podID="9410f552-92c8-41dd-8833-83f9c7a99f1f" containerID="78ccfea6b6471d8735c86673acfd98ea46fe2bb69dbb0ce9fd79ff15fb029e2e" exitCode=0
Nov 02 07:43:32.488332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.485846    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" event={"ID":"9410f552-92c8-41dd-8833-83f9c7a99f1f","Type":"ContainerDied","Data":"78ccfea6b6471d8735c86673acfd98ea46fe2bb69dbb0ce9fd79ff15fb029e2e"}
Nov 02 07:43:32.636583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.636526    4515 scope.go:122] "RemoveContainer" containerID="38a8b0de284adbf5820cd75a5c348f8da34d1e381ba65c9c65ab4a482c3e1d4f"
Nov 02 07:43:32.767598 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.765817    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:43:32.817346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.814569    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-ghjfk"]
Nov 02 07:43:32.868348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.862734    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:43:32.880340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.879050    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-c9cn2"]
Nov 02 07:43:32.931966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:32.931938    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:43:33.014200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.014162    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\") pod \"9410f552-92c8-41dd-8833-83f9c7a99f1f\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") "
Nov 02 07:43:33.014405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.014275    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\") pod \"9410f552-92c8-41dd-8833-83f9c7a99f1f\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") "
Nov 02 07:43:33.014405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.014349    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\") pod \"9410f552-92c8-41dd-8833-83f9c7a99f1f\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") "
Nov 02 07:43:33.014604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.014582    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\" (UniqueName: \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\") pod \"9410f552-92c8-41dd-8833-83f9c7a99f1f\" (UID: \"9410f552-92c8-41dd-8833-83f9c7a99f1f\") "
Nov 02 07:43:33.015038 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.015007    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca" pod "9410f552-92c8-41dd-8833-83f9c7a99f1f" (UID: "9410f552-92c8-41dd-8833-83f9c7a99f1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:33.015220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.015200    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9410f552-92c8-41dd-8833-83f9c7a99f1f-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:33.017259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.017236    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls" pod "9410f552-92c8-41dd-8833-83f9c7a99f1f" (UID: "9410f552-92c8-41dd-8833-83f9c7a99f1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:43:33.022732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.022703    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz" pod "9410f552-92c8-41dd-8833-83f9c7a99f1f" (UID: "9410f552-92c8-41dd-8833-83f9c7a99f1f"). InnerVolumeSpecName "kube-api-access-s7vvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:33.024979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.024055    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs" pod "9410f552-92c8-41dd-8833-83f9c7a99f1f" (UID: "9410f552-92c8-41dd-8833-83f9c7a99f1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:43:33.111829 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.111753    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:43:33.116845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.116815    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7vvz\" (UniqueName: \"kubernetes.io/projected/9410f552-92c8-41dd-8833-83f9c7a99f1f-kube-api-access-s7vvz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:33.116845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.116844    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:33.117008 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.116857    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9410f552-92c8-41dd-8833-83f9c7a99f1f-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:33.374251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.374169    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" path="/var/lib/kubelet/pods/0970f1e5-5b88-44b2-a9dd-8f6266da5a34/volumes"
Nov 02 07:43:33.374705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.374690    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" path="/var/lib/kubelet/pods/5254b663-d289-4fc1-abcf-ebd589039ee9/volumes"
Nov 02 07:43:33.511727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.511687    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm" event={"ID":"9410f552-92c8-41dd-8833-83f9c7a99f1f","Type":"ContainerDied","Data":"370108d416b4dc5fbe4defba678c10eb9603c130c424befd60fe4686a71bcf01"}
Nov 02 07:43:33.512017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.511995    4515 scope.go:122] "RemoveContainer" containerID="78ccfea6b6471d8735c86673acfd98ea46fe2bb69dbb0ce9fd79ff15fb029e2e"
Nov 02 07:43:33.512268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.512250    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"
Nov 02 07:43:33.609716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.609688    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:43:33.617343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:33.616780    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-lcqhm"]
Nov 02 07:43:34.204347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:34.204293    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:43:35.195897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.195871    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:43:35.387791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.387762    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9410f552-92c8-41dd-8833-83f9c7a99f1f" path="/var/lib/kubelet/pods/9410f552-92c8-41dd-8833-83f9c7a99f1f/volumes"
Nov 02 07:43:35.553799 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.553758    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:43:35.555270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555219    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing"
Nov 02 07:43:35.555475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555459    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing"
Nov 02 07:43:35.555592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555582    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9410f552-92c8-41dd-8833-83f9c7a99f1f" containerName="maas-api"
Nov 02 07:43:35.555672 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555662    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9410f552-92c8-41dd-8833-83f9c7a99f1f" containerName="maas-api"
Nov 02 07:43:35.555777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555766    4515 cpu_manager.go:414] "containerMap: removing container" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.555859 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555849    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.555944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.555935    4515 cpu_manager.go:414] "containerMap: removing container" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.556049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.556039    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.556134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.556124    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing"
Nov 02 07:43:35.556273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.556261    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing"
Nov 02 07:43:35.556406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.556393    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5558072d-db71-433e-80a8-a178a6fcaab0" containerName="payload-processing"
Nov 02 07:43:35.557352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557304    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5558072d-db71-433e-80a8-a178a6fcaab0" containerName="payload-processing"
Nov 02 07:43:35.557546 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557530    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.557621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557548    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fc34ba-3dc6-4c8d-9dbb-b62a18064652" containerName="istio-proxy"
Nov 02 07:43:35.557621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557558    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0970f1e5-5b88-44b2-a9dd-8f6266da5a34" containerName="payload-processing"
Nov 02 07:43:35.557621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557567    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5558072d-db71-433e-80a8-a178a6fcaab0" containerName="payload-processing"
Nov 02 07:43:35.557621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557580    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5254b663-d289-4fc1-abcf-ebd589039ee9" containerName="payload-pre-processing"
Nov 02 07:43:35.557621 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.557592    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9410f552-92c8-41dd-8833-83f9c7a99f1f" containerName="maas-api"
Nov 02 07:43:35.563733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.563710    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.584092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.584065    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7e07c6-serving-cert\""
Nov 02 07:43:35.609488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.609450    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:43:35.670971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.670933    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.671124 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.671060    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.671124 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.671094    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzlpd\" (UniqueName: \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.671199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.671176    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.773066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.773025    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.773254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.773197    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.773254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.773228    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzlpd\" (UniqueName: \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.773380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.773260    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.773898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.773872    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.775682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.775650    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.775777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.775690    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.814975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.814758    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzlpd\" (UniqueName: \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\") pod \"maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:35.886639 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:35.886583    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:36.265356 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:36.265323    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f44f08_ce08_4d84_a906_d6de952d6e2c.slice/crio-4058f86c4a3d610603f6f48d71f6a46c7194f5153706ed324a6b5742085359fc WatchSource:0}: Error finding container 4058f86c4a3d610603f6f48d71f6a46c7194f5153706ed324a6b5742085359fc: Status 404 returned error can't find the container with id 4058f86c4a3d610603f6f48d71f6a46c7194f5153706ed324a6b5742085359fc
Nov 02 07:43:36.266616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:36.266595    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:43:36.376857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:36.376826    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:43:36.571968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:36.571869    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" event={"ID":"98f44f08-ce08-4d84-a906-d6de952d6e2c","Type":"ContainerStarted","Data":"4058f86c4a3d610603f6f48d71f6a46c7194f5153706ed324a6b5742085359fc"}
Nov 02 07:43:36.780233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:36.780188    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:43:36.780499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:36.780469    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" containerID="cri-o://c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8" gracePeriod=30
Nov 02 07:43:37.591646 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:37.591606    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" event={"ID":"98f44f08-ce08-4d84-a906-d6de952d6e2c","Type":"ContainerStarted","Data":"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"}
Nov 02 07:43:37.592104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:37.591696    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:37.640071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:37.640020    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" podStartSLOduration=2.298195438 podStartE2EDuration="2.640005196s" totalImagesPullingTime="341.809758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:37.639034782 +0000 UTC m=+1162.401471688" watchObservedRunningTime="2026-11-02 07:43:37.640005196 +0000 UTC m=+1162.402442127"
Nov 02 07:43:38.099059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.099022    4515 patch_prober.go:29] interesting pod/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:43:38.099241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.099077    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:43:38.112184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.112151    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" podUID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" containerName="payload-processing" containerID="cri-o://5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb" gracePeriod=25
Nov 02 07:43:38.488902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.488876    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:43:38.534926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.534894    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\") pod \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") "
Nov 02 07:43:38.535163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.535146    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\" (UniqueName: \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\") pod \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\" (UID: \"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd\") "
Nov 02 07:43:38.535293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.535266    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume" pod "2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" (UID: "2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:38.535637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.535618    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:38.537263 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.537241    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq" pod "2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" (UID: "2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd"). InnerVolumeSpecName "kube-api-access-7dzgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:38.606435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.606247    4515 generic.go:356] "Generic (PLEG): container finished" podID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" containerID="5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb" exitCode=0
Nov 02 07:43:38.607503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.606686    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"
Nov 02 07:43:38.609435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.609407    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" event={"ID":"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd","Type":"ContainerDied","Data":"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"}
Nov 02 07:43:38.609817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.609796    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp" event={"ID":"2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd","Type":"ContainerDied","Data":"e7cf72a49224a0ca7df95461582955c639a3ca9b2791536ea7c0b4c4c3ec02cb"}
Nov 02 07:43:38.609973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.609875    4515 scope.go:122] "RemoveContainer" containerID="5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"
Nov 02 07:43:38.621387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.621363    4515 scope.go:122] "RemoveContainer" containerID="5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"
Nov 02 07:43:38.621725 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:38.621699    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb\": container with ID starting with 5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb not found: ID does not exist" containerID="5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"
Nov 02 07:43:38.621806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.621738    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb"} err="failed to get container status \"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb\": rpc error: code = NotFound desc = could not find container \"5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb\": container with ID starting with 5024627884e5c8ea753026f6f85a234dc38d651c779a73b0971f6046166faffb not found: ID does not exist"
Nov 02 07:43:38.637892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.637852    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzgq\" (UniqueName: \"kubernetes.io/projected/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd-kube-api-access-7dzgq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:38.687426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.687390    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:43:38.693977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:38.693947    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3e5f7-8675875845-58wlp"]
Nov 02 07:43:39.374356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:39.374295    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" path="/var/lib/kubelet/pods/2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd/volumes"
Nov 02 07:43:42.034396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.032520    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:43:42.034396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.033608    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" containerName="payload-processing"
Nov 02 07:43:42.034396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.033626    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" containerName="payload-processing"
Nov 02 07:43:42.034396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.033773    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a477c4c-ff3c-4be7-8bd6-8a50c5798bcd" containerName="payload-processing"
Nov 02 07:43:42.049648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.049614    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.085406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.071380    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-77dc65fb\""
Nov 02 07:43:42.085406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.079532    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:43:42.085895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.085852    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-77dc65fb-dockercfg-wcs5r\""
Nov 02 07:43:42.243357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.227274    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhhh\" (UniqueName: \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.284508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.284470    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.306613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.279375    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:43:42.341283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.336650    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:43:42.341283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.336810    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.434972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.434935    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.435152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.434999    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhhh\" (UniqueName: \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.435152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.435086    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.435284 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.435260    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwjt\" (UniqueName: \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.435980 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.435955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.463238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.463203    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhhh\" (UniqueName: \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\") pod \"payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.514105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.514076    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:43:42.536786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.536754    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwjt\" (UniqueName: \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.536954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.536917    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.537654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.537634    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.573803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.573772    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwjt\" (UniqueName: \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\") pod \"payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") " pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.632973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.632880    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:42.638418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638388    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638485    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638623    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638659    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638798    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638832    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.638947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638884    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.639002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638949    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.639002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.638982    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:42.639104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639015    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\") pod \"21f0daa4-2b52-402b-b253-4c774aa921e9\" (UID: \"21f0daa4-2b52-402b-b253-4c774aa921e9\") "
Nov 02 07:43:42.639208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639187    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:42.639526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639496    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:42.639738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639716    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:42.639950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639917    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.639950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639942    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f0daa4-2b52-402b-b253-4c774aa921e9-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.640143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639957    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.640143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.639970    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.640143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.640089    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:42.641864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.641828    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "kube-api-access-v4855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:42.643771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.643699    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:42.645046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.645024    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:42.645429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.645402    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo" pod "21f0daa4-2b52-402b-b253-4c774aa921e9" (UID: "21f0daa4-2b52-402b-b253-4c774aa921e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:43:42.663747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.663592    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:42.747126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.742899    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.747126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.742933    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.747126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.742946    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.747126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.742961    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f0daa4-2b52-402b-b253-4c774aa921e9-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.747126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.742974    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4855\" (UniqueName: \"kubernetes.io/projected/21f0daa4-2b52-402b-b253-4c774aa921e9-kube-api-access-v4855\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:43:42.874027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.873928    4515 generic.go:356] "Generic (PLEG): container finished" podID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerID="c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8" exitCode=0
Nov 02 07:43:42.874225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.874021    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerDied","Data":"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"}
Nov 02 07:43:42.874225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.874127    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj" event={"ID":"21f0daa4-2b52-402b-b253-4c774aa921e9","Type":"ContainerDied","Data":"942e47e7fb11ed89a94a578ed230d1b1b8ee0ad3cf844260e47242bf628b1e37"}
Nov 02 07:43:42.874225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.874179    4515 scope.go:122] "RemoveContainer" containerID="c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"
Nov 02 07:43:42.876403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.875696    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"
Nov 02 07:43:42.889843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.889388    4515 scope.go:122] "RemoveContainer" containerID="8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"
Nov 02 07:43:42.913137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.913115    4515 scope.go:122] "RemoveContainer" containerID="c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"
Nov 02 07:43:42.913577 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:42.913552    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8\": container with ID starting with c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8 not found: ID does not exist" containerID="c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"
Nov 02 07:43:42.913700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.913581    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8"} err="failed to get container status \"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8\": rpc error: code = NotFound desc = could not find container \"c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8\": container with ID starting with c02f64dabf91c8ef2a86b146c75278f72a434d64982144e35832fdaa04a47ed8 not found: ID does not exist"
Nov 02 07:43:42.913700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.913600    4515 scope.go:122] "RemoveContainer" containerID="8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"
Nov 02 07:43:42.913932 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:42.913904    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e\": container with ID starting with 8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e not found: ID does not exist" containerID="8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"
Nov 02 07:43:42.913997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.913942    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e"} err="failed to get container status \"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e\": rpc error: code = NotFound desc = could not find container \"8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e\": container with ID starting with 8ee16948e726e1a65580bf2541ea8037a106d37761a628b6c421c6c627d98c7e not found: ID does not exist"
Nov 02 07:43:42.962508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.962474    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:43:42.997539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:42.997508    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cc2bcd-openshift-default-d74c5c589-nnpkj"]
Nov 02 07:43:43.005924 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:43.005893    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6994160c_12b1_401a_8acf_9912066465e0.slice/crio-e7f60065d1e1f1dee21d0bcba4f60f4204ebd03215e9a85fd97f184a0882cc63 WatchSource:0}: Error finding container e7f60065d1e1f1dee21d0bcba4f60f4204ebd03215e9a85fd97f184a0882cc63: Status 404 returned error can't find the container with id e7f60065d1e1f1dee21d0bcba4f60f4204ebd03215e9a85fd97f184a0882cc63
Nov 02 07:43:43.010884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.010845    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:43:43.138259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.138212    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:43:43.389965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.389929    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" path="/var/lib/kubelet/pods/21f0daa4-2b52-402b-b253-4c774aa921e9/volumes"
Nov 02 07:43:43.672889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.671488    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:43:43.885636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.885592    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" event={"ID":"e623b172-d2ff-44e7-8270-1a2e6a938f3c","Type":"ContainerStarted","Data":"8a42773634f612bf5606325d558ace9ef85679acf9b1adcc28d89be2f59da706"}
Nov 02 07:43:43.885813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.885662    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" event={"ID":"e623b172-d2ff-44e7-8270-1a2e6a938f3c","Type":"ContainerStarted","Data":"376bedbf182ed0fb534a098b5939e4f0899c2a2a13314423269e641f9d071acc"}
Nov 02 07:43:43.885906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.885821    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:43.887500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.887475    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" event={"ID":"6994160c-12b1-401a-8acf-9912066465e0","Type":"ContainerStarted","Data":"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"}
Nov 02 07:43:43.887643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.887522    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" event={"ID":"6994160c-12b1-401a-8acf-9912066465e0","Type":"ContainerStarted","Data":"e7f60065d1e1f1dee21d0bcba4f60f4204ebd03215e9a85fd97f184a0882cc63"}
Nov 02 07:43:43.887643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.887556    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:43.929962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.929857    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" podStartSLOduration=1.929842509 podStartE2EDuration="1.929842509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:43.928236404 +0000 UTC m=+1168.690673310" watchObservedRunningTime="2026-11-02 07:43:43.929842509 +0000 UTC m=+1168.692279436"
Nov 02 07:43:43.972783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:43.972733    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" podStartSLOduration=2.972719384 podStartE2EDuration="2.972719384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:43.970296344 +0000 UTC m=+1168.732733252" watchObservedRunningTime="2026-11-02 07:43:43.972719384 +0000 UTC m=+1168.735156290"
Nov 02 07:43:46.850715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.850677    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:43:46.851666 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.851638    4515 cpu_manager.go:414] "containerMap: removing container" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:46.851666 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.851664    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:46.851979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.851961    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:46.856361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.856339    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.890552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.885829    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-523de0-openshift-default-dockercfg-ptktx\""
Nov 02 07:43:46.891767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.891740    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940638    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940711    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940749    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcddl\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940780    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940882    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.940945    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.941005    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.941065    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:46.942339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:46.941094    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044178    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcddl\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044230    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044376    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044445    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044509    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044573    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044601    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044711    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.044766    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.046018    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.047085    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.047997    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.048015    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.049330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.048280    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.074336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.050803    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.116383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.102080    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.116383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.104964    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.124424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.122982    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcddl\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.173442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.173231    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:47.673664 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:47.673627    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda05da06_700a_48ab_ae4a_c8849d2481ae.slice/crio-b744ce4a4ec9ded83ff32d2ed9fd64e712d74b4cfe87dfe45d9f861a7ce488ef WatchSource:0}: Error finding container b744ce4a4ec9ded83ff32d2ed9fd64e712d74b4cfe87dfe45d9f861a7ce488ef: Status 404 returned error can't find the container with id b744ce4a4ec9ded83ff32d2ed9fd64e712d74b4cfe87dfe45d9f861a7ce488ef
Nov 02 07:43:47.674796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.674677    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:43:47.677739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.677702    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:47.677816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.677776    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:47.677816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.677808    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:47.728280 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:43:47.728251    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:43:47.924597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.924496    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerStarted","Data":"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"}
Nov 02 07:43:47.924597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.924564    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerStarted","Data":"b744ce4a4ec9ded83ff32d2ed9fd64e712d74b4cfe87dfe45d9f861a7ce488ef"}
Nov 02 07:43:47.979820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:47.979769    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podStartSLOduration=1.979754907 podStartE2EDuration="1.979754907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:47.975211757 +0000 UTC m=+1172.737648665" watchObservedRunningTime="2026-11-02 07:43:47.979754907 +0000 UTC m=+1172.742191813"
Nov 02 07:43:48.173711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:48.173673    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:48.191060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:48.190979    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:48.931172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:48.931142    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:48.932405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:48.932383    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:43:53.107949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.107918    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:43:53.108595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.108580    4515 cpu_manager.go:414] "containerMap: removing container" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:53.108650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.108596    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:53.108694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.108685    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f0daa4-2b52-402b-b253-4c774aa921e9" containerName="istio-proxy"
Nov 02 07:43:53.113392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.113365    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.118192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.118168    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:43:53.122791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.122773    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.135588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.135565    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-523de0\""
Nov 02 07:43:53.139352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.139110    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:43:53.145720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.145692    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-523de0-dockercfg-ntxxp\""
Nov 02 07:43:53.151633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.151607    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:43:53.162364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.162333    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.162481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.162371    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.162481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.162400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkrc\" (UniqueName: \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.162765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.162734    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsvm\" (UniqueName: \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.280332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.280218    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.280332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.280273    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.280332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.280302    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkrc\" (UniqueName: \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.280698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.280619    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsvm\" (UniqueName: \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.290338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.282203    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.290338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.282999    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.350635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.350562    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkrc\" (UniqueName: \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\") pod \"payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.365380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.354278    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsvm\" (UniqueName: \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\") pod \"payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.431617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.431581    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:53.444797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.442235    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:53.833059 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:53.833029    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b16e538_eb9b_4347_86e5_3e66fc9eec3c.slice/crio-c7669a7c2c595af52e8205e7004d59b624bddef5641e9eb7735c3519d056e364 WatchSource:0}: Error finding container c7669a7c2c595af52e8205e7004d59b624bddef5641e9eb7735c3519d056e364: Status 404 returned error can't find the container with id c7669a7c2c595af52e8205e7004d59b624bddef5641e9eb7735c3519d056e364
Nov 02 07:43:53.834817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.834792    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:43:53.874491 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:53.874434    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609fc1d5_d12a_4d0d_9916_a810c2c0d839.slice/crio-1fef551cf9b8469415cf8f1406f2563186cbb73665da6e69f0c63dc98f811fd6 WatchSource:0}: Error finding container 1fef551cf9b8469415cf8f1406f2563186cbb73665da6e69f0c63dc98f811fd6: Status 404 returned error can't find the container with id 1fef551cf9b8469415cf8f1406f2563186cbb73665da6e69f0c63dc98f811fd6
Nov 02 07:43:53.876684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:53.876582    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:43:54.075236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.075059    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" event={"ID":"609fc1d5-d12a-4d0d-9916-a810c2c0d839","Type":"ContainerStarted","Data":"1fef551cf9b8469415cf8f1406f2563186cbb73665da6e69f0c63dc98f811fd6"}
Nov 02 07:43:54.075236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.075170    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:43:54.077364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.077296    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" event={"ID":"1b16e538-eb9b-4347-86e5-3e66fc9eec3c","Type":"ContainerStarted","Data":"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"}
Nov 02 07:43:54.077479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.077378    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:43:54.077479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.077394    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" event={"ID":"1b16e538-eb9b-4347-86e5-3e66fc9eec3c","Type":"ContainerStarted","Data":"c7669a7c2c595af52e8205e7004d59b624bddef5641e9eb7735c3519d056e364"}
Nov 02 07:43:54.120233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.120180    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" podStartSLOduration=1.120168316 podStartE2EDuration="1.120168316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:54.116714436 +0000 UTC m=+1178.879151342" watchObservedRunningTime="2026-11-02 07:43:54.120168316 +0000 UTC m=+1178.882605222"
Nov 02 07:43:54.162377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.162258    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" podStartSLOduration=1.162244716 podStartE2EDuration="1.162244716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:54.159407707 +0000 UTC m=+1178.921844612" watchObservedRunningTime="2026-11-02 07:43:54.162244716 +0000 UTC m=+1178.924681630"
Nov 02 07:43:54.725864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.725829    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:43:54.731606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.731575    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.785661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.785629    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:43:54.791369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.791343    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6cc3f7-openshift-default-dockercfg-hxfmh\""
Nov 02 07:43:54.844567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844532    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.844764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844569    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.844764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844687    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.844764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844740    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.844891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844838    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k476v\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.844891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.844871    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.845022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.845004    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.845113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.845056    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.845165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.845134    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.897598 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.897567    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:43:54.897811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.897799    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:43:54.946811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.946768    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.946811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.946812    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.946872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.946938    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.946992    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k476v\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947019    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947126    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947174    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947224    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947347    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947378    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947782    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.947934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947847    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.948014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.947987    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.949303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.949278    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:54.949585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.949565    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:55.001815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:54.999935    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:55.332573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.332438    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" event={"ID":"609fc1d5-d12a-4d0d-9916-a810c2c0d839","Type":"ContainerStarted","Data":"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"}
Nov 02 07:43:55.337226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.337198    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k476v\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\") pod \"e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") " pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:55.351192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.351159    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:55.678152 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:43:55.678119    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43345e75_8a66_4e3e_8fb1_bfce68636bed.slice/crio-f3d883fff740832dc10732f75b1751f9a8b43417701050b614f4ee57d373b86d WatchSource:0}: Error finding container f3d883fff740832dc10732f75b1751f9a8b43417701050b614f4ee57d373b86d: Status 404 returned error can't find the container with id f3d883fff740832dc10732f75b1751f9a8b43417701050b614f4ee57d373b86d
Nov 02 07:43:55.680965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.680930    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:55.681075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.680996    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:55.681075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.681025    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:55.681418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:55.681388    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:43:56.338465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:56.338422    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerStarted","Data":"306be1d1973053077e37e72c2565a45d0bbef94211835b69d08c0b8fa49e77dc"}
Nov 02 07:43:56.338897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:56.338487    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerStarted","Data":"f3d883fff740832dc10732f75b1751f9a8b43417701050b614f4ee57d373b86d"}
Nov 02 07:43:56.352058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:56.352029    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:56.358366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:56.358338    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:56.410923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:56.410866    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podStartSLOduration=2.4108535460000002 podStartE2EDuration="2.410853546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:56.408557195 +0000 UTC m=+1181.170994105" watchObservedRunningTime="2026-11-02 07:43:56.410853546 +0000 UTC m=+1181.173290451"
Nov 02 07:43:57.345107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:57.345078    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:43:57.346223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:43:57.346200    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:44:01.566730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.566688    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:44:01.647387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.647346    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:44:01.947689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.947593    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:44:01.953521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.953498    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.970745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.970714    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:44:01.977661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.977636    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d12c2-openshift-default-dockercfg-qzg7x\""
Nov 02 07:44:01.979155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979123    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979180    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979298    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lncgh\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979414    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979460    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979528    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979550    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979635    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:01.979736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:01.979683    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.085508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.081443    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.085508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.081488    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.257610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.257572    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.258012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.257985    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.258211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.258193    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.258424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.258406    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.258584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.258568    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.292331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.258782    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncgh\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.292781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.292747    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.292987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.292967    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.293243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.272750    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.293405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.271867    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.294024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.272099    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.314336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.294473    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.364370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.363788    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.364370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.364292    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.379339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.379147    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lncgh\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.454336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.410550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\") pod \"e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") " pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:02.570385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:02.570106    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:03.558460 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:03.558423    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba2b299_c7d4_4e67_b78e_16cb9f97c7cd.slice/crio-5ee23306fb50be9161d3454371f9c9625780fd0e96bb21e6519a85503e6b5aad WatchSource:0}: Error finding container 5ee23306fb50be9161d3454371f9c9625780fd0e96bb21e6519a85503e6b5aad: Status 404 returned error can't find the container with id 5ee23306fb50be9161d3454371f9c9625780fd0e96bb21e6519a85503e6b5aad
Nov 02 07:44:03.559492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:03.559465    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:44:03.561202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:03.561159    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:03.561283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:03.561234    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:03.561283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:03.561263    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:03.655224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:03.655190    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerStarted","Data":"5ee23306fb50be9161d3454371f9c9625780fd0e96bb21e6519a85503e6b5aad"}
Nov 02 07:44:04.666298 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:04.666265    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerStarted","Data":"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"}
Nov 02 07:44:04.742140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:04.742072    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podStartSLOduration=3.742054096 podStartE2EDuration="3.742054096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:04.74166185 +0000 UTC m=+1189.504098756" watchObservedRunningTime="2026-11-02 07:44:04.742054096 +0000 UTC m=+1189.504491002"
Nov 02 07:44:05.341793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:05.341765    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:44:05.572164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:05.572131    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:05.577078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:05.577054    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:05.682083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:05.681995    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:05.684528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:05.684500    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:44:06.345327 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.345262    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:44:06.567614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.567575    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" podUID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" containerName="payload-processing" containerID="cri-o://a9ca997fb6090785598028398d1f4c3ab50bc4cf02d024c6ee260784ed1980a0" gracePeriod=25
Nov 02 07:44:06.648738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.648636    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" podUID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" containerName="payload-pre-processing" containerID="cri-o://a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c" gracePeriod=25
Nov 02 07:44:06.689103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.689073    4515 generic.go:356] "Generic (PLEG): container finished" podID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" containerID="a9ca997fb6090785598028398d1f4c3ab50bc4cf02d024c6ee260784ed1980a0" exitCode=0
Nov 02 07:44:06.689536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.689150    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" event={"ID":"212b6fe9-3ece-460c-9f01-2953f3bc4c8c","Type":"ContainerDied","Data":"a9ca997fb6090785598028398d1f4c3ab50bc4cf02d024c6ee260784ed1980a0"}
Nov 02 07:44:06.991746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:06.991717    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:44:07.041391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.041366    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:44:07.079957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.079924    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\") pod \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") "
Nov 02 07:44:07.080142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.079971    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\" (UniqueName: \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\") pod \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\" (UID: \"212b6fe9-3ece-460c-9f01-2953f3bc4c8c\") "
Nov 02 07:44:07.080142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.080050    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\") pod \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") "
Nov 02 07:44:07.080332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.080291    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\" (UniqueName: \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\") pod \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\" (UID: \"48c0ec82-e6dd-4f7c-82b1-f83d6007533a\") "
Nov 02 07:44:07.080976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.080369    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume" pod "212b6fe9-3ece-460c-9f01-2953f3bc4c8c" (UID: "212b6fe9-3ece-460c-9f01-2953f3bc4c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:07.081191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.081160    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume" pod "48c0ec82-e6dd-4f7c-82b1-f83d6007533a" (UID: "48c0ec82-e6dd-4f7c-82b1-f83d6007533a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:07.081426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.081406    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:07.081515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.081433    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:07.083192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.083170    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr" pod "212b6fe9-3ece-460c-9f01-2953f3bc4c8c" (UID: "212b6fe9-3ece-460c-9f01-2953f3bc4c8c"). InnerVolumeSpecName "kube-api-access-wmjmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:07.083517 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.083493    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj" pod "48c0ec82-e6dd-4f7c-82b1-f83d6007533a" (UID: "48c0ec82-e6dd-4f7c-82b1-f83d6007533a"). InnerVolumeSpecName "kube-api-access-zcnqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:07.183127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.183092    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmjmr\" (UniqueName: \"kubernetes.io/projected/212b6fe9-3ece-460c-9f01-2953f3bc4c8c-kube-api-access-wmjmr\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:07.183258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.183132    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnqj\" (UniqueName: \"kubernetes.io/projected/48c0ec82-e6dd-4f7c-82b1-f83d6007533a-kube-api-access-zcnqj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:07.699355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.699254    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"
Nov 02 07:44:07.699355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.699270    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc" event={"ID":"212b6fe9-3ece-460c-9f01-2953f3bc4c8c","Type":"ContainerDied","Data":"36749a8efb12d6e8d11f56106fac40ee4e494acf90c5bb90279dea4b133d870a"}
Nov 02 07:44:07.699850 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.699364    4515 scope.go:122] "RemoveContainer" containerID="a9ca997fb6090785598028398d1f4c3ab50bc4cf02d024c6ee260784ed1980a0"
Nov 02 07:44:07.701032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.701005    4515 generic.go:356] "Generic (PLEG): container finished" podID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" containerID="a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c" exitCode=0
Nov 02 07:44:07.701149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.701049    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"
Nov 02 07:44:07.701149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.701100    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" event={"ID":"48c0ec82-e6dd-4f7c-82b1-f83d6007533a","Type":"ContainerDied","Data":"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"}
Nov 02 07:44:07.701268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.701151    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd" event={"ID":"48c0ec82-e6dd-4f7c-82b1-f83d6007533a","Type":"ContainerDied","Data":"77c0d70d6853d37ae35ffd6fb5aac9932ef5b84fe70767b623a157ff8a5ba211"}
Nov 02 07:44:07.710686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.710670    4515 scope.go:122] "RemoveContainer" containerID="a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"
Nov 02 07:44:07.719702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.719678    4515 scope.go:122] "RemoveContainer" containerID="a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"
Nov 02 07:44:07.720019 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:07.719998    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c\": container with ID starting with a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c not found: ID does not exist" containerID="a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"
Nov 02 07:44:07.720096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.720024    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c"} err="failed to get container status \"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c\": rpc error: code = NotFound desc = could not find container \"a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c\": container with ID starting with a2b4d6860ce97c5b4146abd8a7e27fec33e2219b6b9a23b912e1b8bf249c175c not found: ID does not exist"
Nov 02 07:44:07.774810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.774771    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:44:07.803790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.803758    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-78788df4ff-6p4rd"]
Nov 02 07:44:07.860160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.860121    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:44:07.864638 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:07.864612    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-697fc8977c-8vvmc"]
Nov 02 07:44:09.374140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:09.374107    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" path="/var/lib/kubelet/pods/212b6fe9-3ece-460c-9f01-2953f3bc4c8c/volumes"
Nov 02 07:44:09.374586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:09.374572    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" path="/var/lib/kubelet/pods/48c0ec82-e6dd-4f7c-82b1-f83d6007533a/volumes"
Nov 02 07:44:10.946960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.946912    4515 generic.go:356] "Generic (PLEG): container finished" podID="962061a8-44c3-4e04-8467-9eeab4761cea" containerID="50add28e2b3afcad546bbe170590b1bcc8d78be9f1b99f9f19bae6f4a9741538" exitCode=0
Nov 02 07:44:10.947852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.947813    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerDied","Data":"50add28e2b3afcad546bbe170590b1bcc8d78be9f1b99f9f19bae6f4a9741538"}
Nov 02 07:44:10.948591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.948545    4515 scope.go:122] "RemoveContainer" containerID="50add28e2b3afcad546bbe170590b1bcc8d78be9f1b99f9f19bae6f4a9741538"
Nov 02 07:44:10.950993 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.950949    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:10.951102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.951051    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:10.951167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:10.951135    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:11.956650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:11.956614    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerStarted","Data":"cc672a5e27e2aef22fa00be2384d8177b03b1a385fc8a12c385e9c1d5f7c6b47"}
Nov 02 07:44:12.937742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:12.937713    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:44:12.938084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:12.938062    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 07:44:12.938238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:12.938213    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 07:44:13.955858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:13.955814    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 07:44:13.956463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:13.955896    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 07:44:14.940742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:14.940702    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 07:44:14.941031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:14.940999    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 07:44:15.743866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:15.743834    4515 scope.go:122] "RemoveContainer" containerID="4cca9a0d9f3caaa18deffe4b331d0a288bcd91cadc01f89cb3f9696bfd5b20d7"
Nov 02 07:44:15.940337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:15.938666    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 07:44:15.940337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:15.938724    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 07:44:16.941788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:16.941742    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 07:44:16.942208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:16.941816    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 07:44:17.723784 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:17.723749    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:44:17.938949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:17.938920    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:44:17.939358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:17.939336    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:44:17.940110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:17.940091    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:44:19.955391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.955359    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:44:19.956960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.956936    4515 cpu_manager.go:414] "containerMap: removing container" podUID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" containerName="payload-processing"
Nov 02 07:44:19.956960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.956959    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" containerName="payload-processing"
Nov 02 07:44:19.957140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.956985    4515 cpu_manager.go:414] "containerMap: removing container" podUID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" containerName="payload-pre-processing"
Nov 02 07:44:19.957140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.956993    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" containerName="payload-pre-processing"
Nov 02 07:44:19.957291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.957274    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="212b6fe9-3ece-460c-9f01-2953f3bc4c8c" containerName="payload-processing"
Nov 02 07:44:19.957407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:19.957300    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c0ec82-e6dd-4f7c-82b1-f83d6007533a" containerName="payload-pre-processing"
Nov 02 07:44:20.167684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.167654    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:44:20.184339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.168754    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.326332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.185724    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.326332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.326171    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.424512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.424480    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e07c6\""
Nov 02 07:44:20.522592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.435658    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:44:20.524341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.439600    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.524341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.523093    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.524341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.440625    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.548566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.548527    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") pod \"payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.781241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.781209    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:44:20.785933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.785910    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:20.819688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.819647    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6cc3f7-serving-cert\""
Nov 02 07:44:20.827141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.827112    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:44:20.840109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.840066    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:20.930735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.930702    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:20.930893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.930773    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:20.931062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.931033    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhgp\" (UniqueName: \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:20.931238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:20.931218    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.033031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.032944    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.033486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.033044    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhgp\" (UniqueName: \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.033486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.033151    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.033486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.033352    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.033862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.033830    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.035558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.035537    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.035744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.035721    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.060432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.060407    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhgp\" (UniqueName: \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\") pod \"maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.114330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.111218    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:21.251716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.251676    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:44:21.253744 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:21.253725    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbdfa55_acd6_4821_a64b_bc209ff6e5d2.slice/crio-554b8901f79fbdd8b09cc4fe9664e0ea470393ae3f66f39223bde0bdb2b1db19 WatchSource:0}: Error finding container 554b8901f79fbdd8b09cc4fe9664e0ea470393ae3f66f39223bde0bdb2b1db19: Status 404 returned error can't find the container with id 554b8901f79fbdd8b09cc4fe9664e0ea470393ae3f66f39223bde0bdb2b1db19
Nov 02 07:44:21.556362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.554450    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" event={"ID":"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2","Type":"ContainerStarted","Data":"554b8901f79fbdd8b09cc4fe9664e0ea470393ae3f66f39223bde0bdb2b1db19"}
Nov 02 07:44:21.576342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:21.575546    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:44:21.644554 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:21.644511    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ba0f21_acd5_474e_bc60_c58de68da073.slice/crio-f1d22e35e95f9b958fe649a987e2e54ed1272d606e0746ffe639e1d650362792 WatchSource:0}: Error finding container f1d22e35e95f9b958fe649a987e2e54ed1272d606e0746ffe639e1d650362792: Status 404 returned error can't find the container with id f1d22e35e95f9b958fe649a987e2e54ed1272d606e0746ffe639e1d650362792
Nov 02 07:44:23.083114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.083071    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" event={"ID":"43ba0f21-acd5-474e-bc60-c58de68da073","Type":"ContainerStarted","Data":"cc46d5f46611f6095d8f412050bc7429c09c34552d02295bce72ff6ebf039824"}
Nov 02 07:44:23.090342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.086373    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" event={"ID":"43ba0f21-acd5-474e-bc60-c58de68da073","Type":"ContainerStarted","Data":"f1d22e35e95f9b958fe649a987e2e54ed1272d606e0746ffe639e1d650362792"}
Nov 02 07:44:23.105351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.090645    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:23.175167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.175100    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" podStartSLOduration=2.763604552 podStartE2EDuration="3.17507856s" totalImagesPullingTime="411.474008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:23.168389559 +0000 UTC m=+1207.930826472" watchObservedRunningTime="2026-11-02 07:44:23.17507856 +0000 UTC m=+1207.937515470"
Nov 02 07:44:23.854264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.854225    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-dac468"]
Nov 02 07:44:23.866563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.866521    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-dac468"]
Nov 02 07:44:23.866748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.866651    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:23.882405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.882378    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmvw6\""
Nov 02 07:44:23.962447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:23.962412    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccp8\" (UniqueName: \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\") pod \"test-curl-7355-dac468\" (UID: \"0283b8b0-079f-49d6-a391-9299ee78b751\") " pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:24.063931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:24.063891    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccp8\" (UniqueName: \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\") pod \"test-curl-7355-dac468\" (UID: \"0283b8b0-079f-49d6-a391-9299ee78b751\") " pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:24.088362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:24.088324    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccp8\" (UniqueName: \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\") pod \"test-curl-7355-dac468\" (UID: \"0283b8b0-079f-49d6-a391-9299ee78b751\") " pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:24.185082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:24.185005    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:24.589688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:24.582737    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-dac468"]
Nov 02 07:44:25.101489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:25.100719    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-dac468" event={"ID":"0283b8b0-079f-49d6-a391-9299ee78b751","Type":"ContainerStarted","Data":"a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b"}
Nov 02 07:44:25.101489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:25.100761    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-dac468" event={"ID":"0283b8b0-079f-49d6-a391-9299ee78b751","Type":"ContainerStarted","Data":"2ca5552082ce31d10e3bf78eb81ee9dd3ab5e8b23c0f7123faddd638b3f83522"}
Nov 02 07:44:25.144171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:25.144122    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7355-dac468" podStartSLOduration=2.144106438 podStartE2EDuration="2.144106438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:25.142643682 +0000 UTC m=+1209.905080590" watchObservedRunningTime="2026-11-02 07:44:25.144106438 +0000 UTC m=+1209.906543344"
Nov 02 07:44:26.109286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:26.109247    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" event={"ID":"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2","Type":"ContainerStarted","Data":"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"}
Nov 02 07:44:26.109764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:26.109538    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:26.157837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:26.157802    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-dac468"]
Nov 02 07:44:26.177845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:26.177802    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-dac468"]
Nov 02 07:44:26.178608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:26.178562    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" podStartSLOduration=3.355098568 podStartE2EDuration="7.17854542s" totalImagesPullingTime="3.823446852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:26.172918575 +0000 UTC m=+1210.935355485" watchObservedRunningTime="2026-11-02 07:44:26.17854542 +0000 UTC m=+1210.940982328"
Nov 02 07:44:27.116221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:27.116180    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7355-dac468" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" containerName="test-curl-7355-dac468" containerID="cri-o://a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b" gracePeriod=2
Nov 02 07:44:27.152403 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:27.152358    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7355-dac468\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:28.636118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.636071    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:44:28.636870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.636850    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" containerName="test-curl-7355-dac468"
Nov 02 07:44:28.636870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.636869    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" containerName="test-curl-7355-dac468"
Nov 02 07:44:28.636996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.636968    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" containerName="test-curl-7355-dac468"
Nov 02 07:44:28.640932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.640910    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:28.663719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.663686    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:44:28.670521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.670476    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5a032f\""
Nov 02 07:44:28.670798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.670774    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5a032f-dockercfg-464vh\""
Nov 02 07:44:28.833356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.829005    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:28.833781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.833752    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpsrq\" (UniqueName: \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:28.990995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.990912    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:44:29.038606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.992388    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:29.038606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.037945    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpsrq\" (UniqueName: \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:29.038606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:28.993786    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:29.068769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.067790    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:44:29.068769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.067922    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.089953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.089864    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpsrq\" (UniqueName: \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\") pod \"payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:29.140881 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.140852    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmww\" (UniqueName: \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.141071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.140923    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.243712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.242208    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.243712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.242682    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmww\" (UniqueName: \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.253356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.244686    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.258395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.258363    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:29.273159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.271530    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmww\" (UniqueName: \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\") pod \"payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") " pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.420095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.419908    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:29.440566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.440532    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:29.462362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.462336    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\" (UniqueName: \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\") pod \"0283b8b0-079f-49d6-a391-9299ee78b751\" (UID: \"0283b8b0-079f-49d6-a391-9299ee78b751\") "
Nov 02 07:44:29.465243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.465207    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8" pod "0283b8b0-079f-49d6-a391-9299ee78b751" (UID: "0283b8b0-079f-49d6-a391-9299ee78b751"). InnerVolumeSpecName "kube-api-access-wccp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:29.565391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.565349    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccp8\" (UniqueName: \"kubernetes.io/projected/0283b8b0-079f-49d6-a391-9299ee78b751-kube-api-access-wccp8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:29.631286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.631087    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:44:29.639518 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:29.638747    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8dd368_31dc_4423_8ae8_274b09399d04.slice/crio-d084cddb8e4b5266f3b9aa728f31d0d9c75ab9758221f8487c541ad39e36fe48 WatchSource:0}: Error finding container d084cddb8e4b5266f3b9aa728f31d0d9c75ab9758221f8487c541ad39e36fe48: Status 404 returned error can't find the container with id d084cddb8e4b5266f3b9aa728f31d0d9c75ab9758221f8487c541ad39e36fe48
Nov 02 07:44:29.757634 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:29.757570    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32885759_fe9f_4abd_a145_80c4642b0779.slice/crio-3eacc404104c0a5ec63707c4095027a96bbb37395efe2661764bdaa4aafff289 WatchSource:0}: Error finding container 3eacc404104c0a5ec63707c4095027a96bbb37395efe2661764bdaa4aafff289: Status 404 returned error can't find the container with id 3eacc404104c0a5ec63707c4095027a96bbb37395efe2661764bdaa4aafff289
Nov 02 07:44:29.759053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:29.759027    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:44:30.048140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.048063    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:44:30.052830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.052800    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.072618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.072588    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9d12c2-serving-cert\""
Nov 02 07:44:30.074147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.074120    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:44:30.122096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.122067    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:44:30.156868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.156699    4515 generic.go:356] "Generic (PLEG): container finished" podID="0283b8b0-079f-49d6-a391-9299ee78b751" containerID="a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b" exitCode=137
Nov 02 07:44:30.156868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.156744    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:30.156868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.156801    4515 scope.go:122] "RemoveContainer" containerID="a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b"
Nov 02 07:44:30.159153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.159127    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" event={"ID":"32885759-fe9f-4abd-a145-80c4642b0779","Type":"ContainerStarted","Data":"c66fa8dca4a19f70c7221fea9d1d808df6b2c55d61f2d63af00da693954d5b65"}
Nov 02 07:44:30.159279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.159183    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" event={"ID":"32885759-fe9f-4abd-a145-80c4642b0779","Type":"ContainerStarted","Data":"3eacc404104c0a5ec63707c4095027a96bbb37395efe2661764bdaa4aafff289"}
Nov 02 07:44:30.159526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.159508    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:30.161050 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.161028    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" event={"ID":"fc8dd368-31dc-4423-8ae8-274b09399d04","Type":"ContainerStarted","Data":"5bbd9160c78dbacf635f4b1a5be251998bc6cafca0964825758b4290b93a8f04"}
Nov 02 07:44:30.161161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.161073    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" event={"ID":"fc8dd368-31dc-4423-8ae8-274b09399d04","Type":"ContainerStarted","Data":"d084cddb8e4b5266f3b9aa728f31d0d9c75ab9758221f8487c541ad39e36fe48"}
Nov 02 07:44:30.161161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.161112    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:30.169648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.169624    4515 scope.go:122] "RemoveContainer" containerID="a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b"
Nov 02 07:44:30.170002 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:30.169982    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b\": container with ID starting with a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b not found: ID does not exist" containerID="a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b"
Nov 02 07:44:30.170083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.170012    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b"} err="failed to get container status \"a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b\": rpc error: code = NotFound desc = could not find container \"a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b\": container with ID starting with a41cee915d439f90913973944308b41f0af9e06f5ed3c765e9a1b2b69933a45b not found: ID does not exist"
Nov 02 07:44:30.175805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.175781    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.176064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.176042    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.176112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.176087    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmwk\" (UniqueName: \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.176344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.176321    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.183387 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:30.181946    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7355-dac468\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" pod="opendatahub/test-curl-7355-dac468"
Nov 02 07:44:30.226467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.226416    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" podStartSLOduration=2.226400918 podStartE2EDuration="2.226400918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:30.222722001 +0000 UTC m=+1214.985158909" watchObservedRunningTime="2026-11-02 07:44:30.226400918 +0000 UTC m=+1214.988837824"
Nov 02 07:44:30.266827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.266766    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" podStartSLOduration=2.266751556 podStartE2EDuration="2.266751556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:30.264350557 +0000 UTC m=+1215.026787463" watchObservedRunningTime="2026-11-02 07:44:30.266751556 +0000 UTC m=+1215.029188501"
Nov 02 07:44:30.278177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.278144    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.278394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.278368    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.278624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.278602    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.278721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.278648    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmwk\" (UniqueName: \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.278970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.278945    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.280872 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.280851    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.281272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.281251    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.302172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.302100    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmwk\" (UniqueName: \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\") pod \"maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.367242 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.367209    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:30.859763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:30.859731    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:44:30.860267 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:30.859737    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b90158_0fe7_4b33_b562_d566bb916c17.slice/crio-03745984781c5645e574a5605041dcf8008130a21bb4c5f6a95330254cdc0750 WatchSource:0}: Error finding container 03745984781c5645e574a5605041dcf8008130a21bb4c5f6a95330254cdc0750: Status 404 returned error can't find the container with id 03745984781c5645e574a5605041dcf8008130a21bb4c5f6a95330254cdc0750
Nov 02 07:44:31.179893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:31.179807    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" event={"ID":"59b90158-0fe7-4b33-b562-d566bb916c17","Type":"ContainerStarted","Data":"03745984781c5645e574a5605041dcf8008130a21bb4c5f6a95330254cdc0750"}
Nov 02 07:44:31.404505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:31.404469    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0283b8b0-079f-49d6-a391-9299ee78b751" path="/var/lib/kubelet/pods/0283b8b0-079f-49d6-a391-9299ee78b751/volumes"
Nov 02 07:44:32.139360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.139241    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:44:32.143816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.143794    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.176884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.176849    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:44:32.189802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.189770    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" event={"ID":"59b90158-0fe7-4b33-b562-d566bb916c17","Type":"ContainerStarted","Data":"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"}
Nov 02 07:44:32.189965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.189836    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:32.191468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.191444    4515 generic.go:356] "Generic (PLEG): container finished" podID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerID="4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248" exitCode=0
Nov 02 07:44:32.191592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.191513    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerDied","Data":"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"}
Nov 02 07:44:32.191918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.191901    4515 scope.go:122] "RemoveContainer" containerID="4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"
Nov 02 07:44:32.193694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.193663    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.193797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.193732    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.193797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.193762    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.250706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.250672    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.250844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.250760    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.250844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.250823    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phltv\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.250929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.250863    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.250929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.250889    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.251032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.251020    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.251070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.251042    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.251198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.251178    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.251364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.251342    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.322592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.322529    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" podStartSLOduration=1.9765956139999998 podStartE2EDuration="2.32250717s" totalImagesPullingTime="345.911556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:32.31847139 +0000 UTC m=+1217.080908297" watchObservedRunningTime="2026-11-02 07:44:32.32250717 +0000 UTC m=+1217.084944076"
Nov 02 07:44:32.352874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.352830    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.352912    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.352941    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phltv\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.352967    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.353010    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353284 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.353138    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353284 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.353159    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.353295    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.353513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.353493    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.354292 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.354272    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.354796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.354780    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.360323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.356439    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.360323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.356695    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.360323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.357194    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.360323 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.358244    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.370624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.370596    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.398809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.398736    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.399039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.399014    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phltv\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.461424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.461388    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:32.944477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.944428    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:44:32.945472 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:32.945427    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405fbefd_42c7_48b0_b690_5449dbb6ec71.slice/crio-d6a3f31e52831ac416fa8d7fb39c6c9c412fc3d46901ff466f8d47830637ff41 WatchSource:0}: Error finding container d6a3f31e52831ac416fa8d7fb39c6c9c412fc3d46901ff466f8d47830637ff41: Status 404 returned error can't find the container with id d6a3f31e52831ac416fa8d7fb39c6c9c412fc3d46901ff466f8d47830637ff41
Nov 02 07:44:32.948926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.948878    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.949052 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.949031    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.949109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:32.949085    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:33.201656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.200524    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" event={"ID":"405fbefd-42c7-48b0-b690-5449dbb6ec71","Type":"ContainerStarted","Data":"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"}
Nov 02 07:44:33.201656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.200588    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" event={"ID":"405fbefd-42c7-48b0-b690-5449dbb6ec71","Type":"ContainerStarted","Data":"d6a3f31e52831ac416fa8d7fb39c6c9c412fc3d46901ff466f8d47830637ff41"}
Nov 02 07:44:33.202779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.202747    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerStarted","Data":"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"}
Nov 02 07:44:33.266259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.266206    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podStartSLOduration=1.266191721 podStartE2EDuration="1.266191721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:33.265186918 +0000 UTC m=+1218.027623806" watchObservedRunningTime="2026-11-02 07:44:33.266191721 +0000 UTC m=+1218.028628626"
Nov 02 07:44:33.462033 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.461952    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:33.940335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:33.933421    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:44:34.173895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:34.173816    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:44:34.175274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:34.175244    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:44:34.175430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:34.175292    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 07:44:34.474040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:34.473676    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 07:44:34.474040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:34.473745    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 07:44:35.173981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:35.173941    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:44:35.174169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:35.173997    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 07:44:35.464958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:35.464847    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 07:44:35.464958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:35.464935    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 07:44:36.173788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:36.173755    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:44:36.174299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:36.174267    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 07:44:36.462623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:36.462516    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 07:44:36.462893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:36.462862    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 07:44:37.157555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:37.157511    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:44:37.174786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:37.174745    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:44:37.175289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:37.174805    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 07:44:37.465817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:37.465711    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 07:44:37.465817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:37.465776    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 07:44:38.174123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:38.174091    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:44:38.174383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:38.174362    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 07:44:38.207988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:38.207957    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:44:38.465709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:38.465597    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 07:44:38.465964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:38.465936    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 07:44:39.224974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:39.224939    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:44:39.226988 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:39.226962    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:44:39.706173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:39.706146    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:39.707750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:39.707725    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:39.708901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:39.708878    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:44:41.183000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:41.182968    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:44:41.183432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:41.183042    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:44:41.347683 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:41.347649    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:44:41.403553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:41.403517    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:44:46.348832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.348787    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" podUID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" containerName="payload-processing" containerID="cri-o://6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398" gracePeriod=25
Nov 02 07:44:46.404053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.404000    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" podUID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" containerName="payload-pre-processing" containerID="cri-o://62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d" gracePeriod=25
Nov 02 07:44:46.908756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.908606    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:44:46.971053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.971019    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\" (UniqueName: \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\") pod \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") "
Nov 02 07:44:46.971478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.971433    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\") pod \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\" (UID: \"609fc1d5-d12a-4d0d-9916-a810c2c0d839\") "
Nov 02 07:44:46.980153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.972136    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume" pod "609fc1d5-d12a-4d0d-9916-a810c2c0d839" (UID: "609fc1d5-d12a-4d0d-9916-a810c2c0d839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:46.981826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:46.981782    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm" pod "609fc1d5-d12a-4d0d-9916-a810c2c0d839" (UID: "609fc1d5-d12a-4d0d-9916-a810c2c0d839"). InnerVolumeSpecName "kube-api-access-2nsvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:47.054573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.054543    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:44:47.080741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.080707    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\") pod \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") "
Nov 02 07:44:47.081205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.081152    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume" pod "1b16e538-eb9b-4347-86e5-3e66fc9eec3c" (UID: "1b16e538-eb9b-4347-86e5-3e66fc9eec3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:47.081354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.081330    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\" (UniqueName: \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\") pod \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\" (UID: \"1b16e538-eb9b-4347-86e5-3e66fc9eec3c\") "
Nov 02 07:44:47.082146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.082124    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609fc1d5-d12a-4d0d-9916-a810c2c0d839-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:47.082220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.082154    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:47.082220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.082169    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nsvm\" (UniqueName: \"kubernetes.io/projected/609fc1d5-d12a-4d0d-9916-a810c2c0d839-kube-api-access-2nsvm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:47.084298 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.084273    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc" pod "1b16e538-eb9b-4347-86e5-3e66fc9eec3c" (UID: "1b16e538-eb9b-4347-86e5-3e66fc9eec3c"). InnerVolumeSpecName "kube-api-access-dmkrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:47.183211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.183179    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkrc\" (UniqueName: \"kubernetes.io/projected/1b16e538-eb9b-4347-86e5-3e66fc9eec3c-kube-api-access-dmkrc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:44:47.461786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.461752    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" containerID="62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d" exitCode=0
Nov 02 07:44:47.462271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.461798    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"
Nov 02 07:44:47.462271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.461838    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" event={"ID":"1b16e538-eb9b-4347-86e5-3e66fc9eec3c","Type":"ContainerDied","Data":"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"}
Nov 02 07:44:47.462271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.461901    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl" event={"ID":"1b16e538-eb9b-4347-86e5-3e66fc9eec3c","Type":"ContainerDied","Data":"c7669a7c2c595af52e8205e7004d59b624bddef5641e9eb7735c3519d056e364"}
Nov 02 07:44:47.462271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.461937    4515 scope.go:122] "RemoveContainer" containerID="62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"
Nov 02 07:44:47.463473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.463450    4515 generic.go:356] "Generic (PLEG): container finished" podID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" containerID="6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398" exitCode=0
Nov 02 07:44:47.463588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.463517    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" event={"ID":"609fc1d5-d12a-4d0d-9916-a810c2c0d839","Type":"ContainerDied","Data":"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"}
Nov 02 07:44:47.463588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.463556    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"
Nov 02 07:44:47.463715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.463555    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7" event={"ID":"609fc1d5-d12a-4d0d-9916-a810c2c0d839","Type":"ContainerDied","Data":"1fef551cf9b8469415cf8f1406f2563186cbb73665da6e69f0c63dc98f811fd6"}
Nov 02 07:44:47.472475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.472434    4515 scope.go:122] "RemoveContainer" containerID="62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"
Nov 02 07:44:47.472713 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:47.472696    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d\": container with ID starting with 62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d not found: ID does not exist" containerID="62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"
Nov 02 07:44:47.472761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.472721    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d"} err="failed to get container status \"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d\": rpc error: code = NotFound desc = could not find container \"62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d\": container with ID starting with 62d5c618a1a8be1aa2cae25fd07d65607a9e038c0d65313067f84b0a919c375d not found: ID does not exist"
Nov 02 07:44:47.472761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.472738    4515 scope.go:122] "RemoveContainer" containerID="6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"
Nov 02 07:44:47.481741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.481721    4515 scope.go:122] "RemoveContainer" containerID="6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"
Nov 02 07:44:47.482024 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:47.481996    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398\": container with ID starting with 6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398 not found: ID does not exist" containerID="6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"
Nov 02 07:44:47.482079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.482025    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398"} err="failed to get container status \"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398\": rpc error: code = NotFound desc = could not find container \"6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398\": container with ID starting with 6b5753040b2f5babe5b1b1dbb2b2842c9fd7392fa16d842602aa58018e548398 not found: ID does not exist"
Nov 02 07:44:47.544985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.544948    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:44:47.548910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.548885    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-56988c55f5-kkctl"]
Nov 02 07:44:47.570785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.570754    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:44:47.574802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:47.574776    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-64c99495-bvdn7"]
Nov 02 07:44:47.748048 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:44:47.747963    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:44:49.374834 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:49.374803    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" path="/var/lib/kubelet/pods/1b16e538-eb9b-4347-86e5-3e66fc9eec3c/volumes"
Nov 02 07:44:49.375345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:49.375297    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" path="/var/lib/kubelet/pods/609fc1d5-d12a-4d0d-9916-a810c2c0d839/volumes"
Nov 02 07:44:50.774941 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.774903    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:44:50.775588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775573    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" containerName="payload-pre-processing"
Nov 02 07:44:50.775653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775589    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" containerName="payload-pre-processing"
Nov 02 07:44:50.775653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775603    4515 cpu_manager.go:414] "containerMap: removing container" podUID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" containerName="payload-processing"
Nov 02 07:44:50.775653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775608    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" containerName="payload-processing"
Nov 02 07:44:50.775747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775737    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b16e538-eb9b-4347-86e5-3e66fc9eec3c" containerName="payload-pre-processing"
Nov 02 07:44:50.775780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.775745    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="609fc1d5-d12a-4d0d-9916-a810c2c0d839" containerName="payload-processing"
Nov 02 07:44:50.781224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.781202    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.799020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.798996    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5a032f-serving-cert\""
Nov 02 07:44:50.854358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.837388    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.854358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.837652    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.854358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.837742    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8zm\" (UniqueName: \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.854358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.837837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.855432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.855395    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:44:50.942723 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.942679    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.942908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.942772    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8zm\" (UniqueName: \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.942908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.942862    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.943063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.942943    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.943587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.943558    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.945833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.945693    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.945833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.945719    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:50.967563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:50.967535    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8zm\" (UniqueName: \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\") pod \"maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:51.103383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:51.097595    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:51.745791 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:44:51.745720    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a751671_2b34_4f20_b010_9d907ef59693.slice/crio-f6e9dde8e2317452c765c1fdd142eba74db58d34e8a39fd77d83432d147eae0e WatchSource:0}: Error finding container f6e9dde8e2317452c765c1fdd142eba74db58d34e8a39fd77d83432d147eae0e: Status 404 returned error can't find the container with id f6e9dde8e2317452c765c1fdd142eba74db58d34e8a39fd77d83432d147eae0e
Nov 02 07:44:51.748520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:51.748497    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:44:52.583069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:52.583035    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" event={"ID":"5a751671-2b34-4f20-b010-9d907ef59693","Type":"ContainerStarted","Data":"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"}
Nov 02 07:44:52.583483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:52.583095    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:52.583483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:52.583112    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" event={"ID":"5a751671-2b34-4f20-b010-9d907ef59693","Type":"ContainerStarted","Data":"f6e9dde8e2317452c765c1fdd142eba74db58d34e8a39fd77d83432d147eae0e"}
Nov 02 07:44:52.630931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:52.630836    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" podStartSLOduration=2.282461722 podStartE2EDuration="2.63081637s" totalImagesPullingTime="348.354648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:52.627103829 +0000 UTC m=+1237.389540736" watchObservedRunningTime="2026-11-02 07:44:52.63081637 +0000 UTC m=+1237.393253277"
Nov 02 07:44:56.619570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.619533    4515 generic.go:356] "Generic (PLEG): container finished" podID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerID="306be1d1973053077e37e72c2565a45d0bbef94211835b69d08c0b8fa49e77dc" exitCode=0
Nov 02 07:44:56.620021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.619602    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerDied","Data":"306be1d1973053077e37e72c2565a45d0bbef94211835b69d08c0b8fa49e77dc"}
Nov 02 07:44:56.620083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.620069    4515 scope.go:122] "RemoveContainer" containerID="306be1d1973053077e37e72c2565a45d0bbef94211835b69d08c0b8fa49e77dc"
Nov 02 07:44:56.622000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.621968    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:56.622100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.622040    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:56.622100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:56.622072    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:57.345593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:57.345553    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:44:57.649783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:57.649694    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerStarted","Data":"116947f249777cfce8bf137e88b561364411af7e6f445da9835b4785be1b9b36"}
Nov 02 07:44:58.352693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:58.352636    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:44:58.354000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:58.353966    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 07:44:58.354161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:58.354039    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 07:44:58.623386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:58.616497    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:44:59.353334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:59.352794    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 07:44:59.353334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:44:59.352870    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 07:45:00.187736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.187569    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:00.200346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.200216    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.213656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.213626    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:00.215026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.214990    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.215163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.215087    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqns\" (UniqueName: \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.263593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.263562    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.283784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.283757    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:45:00.284797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.284403    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:00.289962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.289545    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:00.335664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.334118    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwhz\" (UniqueName: \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.335664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.334214    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.335664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.334263    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.335664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.334357    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqns\" (UniqueName: \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.335664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.335604    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.355870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.353873    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 07:45:00.355870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.353932    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 07:45:00.376218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.376163    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqns\" (UniqueName: \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\") pod \"maas-api-key-cleanup-29893425-g9jq2\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.454388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.453540    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwhz\" (UniqueName: \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.454388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.453634    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.464383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.461139    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.483862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.483828    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwhz\" (UniqueName: \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\") pod \"maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.575084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.575049    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:00.596378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.596351    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:00.978368 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:00.975210    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d27ce22_1af0_418e_b5e6_1e19a0615143.slice/crio-6edcd9df19aa64fc4f38f7d53bdf58a1bebf8559cec268204de0cc5c1d5e0d5d WatchSource:0}: Error finding container 6edcd9df19aa64fc4f38f7d53bdf58a1bebf8559cec268204de0cc5c1d5e0d5d: Status 404 returned error can't find the container with id 6edcd9df19aa64fc4f38f7d53bdf58a1bebf8559cec268204de0cc5c1d5e0d5d
Nov 02 07:45:00.979379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:00.979337    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:01.072828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.072779    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:01.072997 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:01.072845    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f45e798_178a_45a3_b779_29548b7afae8.slice/crio-eb2e4a2be3e860c6db09b3e0116d69accf8161242cfc577bde16cf17851eeab7 WatchSource:0}: Error finding container eb2e4a2be3e860c6db09b3e0116d69accf8161242cfc577bde16cf17851eeab7: Status 404 returned error can't find the container with id eb2e4a2be3e860c6db09b3e0116d69accf8161242cfc577bde16cf17851eeab7
Nov 02 07:45:01.360733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.357437    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 07:45:01.360733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.357500    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 07:45:01.775612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.773287    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerStarted","Data":"eb4751e0ebda44e5bbd74e321f8b78435cae6b9541510639fd935b874431f294"}
Nov 02 07:45:01.775612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.773369    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerStarted","Data":"6edcd9df19aa64fc4f38f7d53bdf58a1bebf8559cec268204de0cc5c1d5e0d5d"}
Nov 02 07:45:01.776909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.776847    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerStarted","Data":"c1ed295ae538458eedee1f4f6d483a0ad4936f8d76945e001e388b0c6a659b31"}
Nov 02 07:45:01.776909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.776907    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerStarted","Data":"eb2e4a2be3e860c6db09b3e0116d69accf8161242cfc577bde16cf17851eeab7"}
Nov 02 07:45:01.867336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.866320    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" podStartSLOduration=1.86628365 podStartE2EDuration="1.86628365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:01.838142645 +0000 UTC m=+1246.600579553" watchObservedRunningTime="2026-11-02 07:45:01.86628365 +0000 UTC m=+1246.628720557"
Nov 02 07:45:01.894273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:01.894215    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" podStartSLOduration=1.89419958 podStartE2EDuration="1.89419958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:01.890178061 +0000 UTC m=+1246.652614993" watchObservedRunningTime="2026-11-02 07:45:01.89419958 +0000 UTC m=+1246.656636485"
Nov 02 07:45:02.371813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.359948    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 07:45:02.371813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.360005    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 07:45:02.785705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.785672    4515 generic.go:356] "Generic (PLEG): container finished" podID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerID="eb4751e0ebda44e5bbd74e321f8b78435cae6b9541510639fd935b874431f294" exitCode=60
Nov 02 07:45:02.786160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.786143    4515 scope.go:122] "RemoveContainer" containerID="eb4751e0ebda44e5bbd74e321f8b78435cae6b9541510639fd935b874431f294"
Nov 02 07:45:02.786489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.786446    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerDied","Data":"eb4751e0ebda44e5bbd74e321f8b78435cae6b9541510639fd935b874431f294"}
Nov 02 07:45:02.788590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.788183    4515 generic.go:356] "Generic (PLEG): container finished" podID="4f45e798-178a-45a3-b779-29548b7afae8" containerID="c1ed295ae538458eedee1f4f6d483a0ad4936f8d76945e001e388b0c6a659b31" exitCode=60
Nov 02 07:45:02.788590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.788300    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerDied","Data":"c1ed295ae538458eedee1f4f6d483a0ad4936f8d76945e001e388b0c6a659b31"}
Nov 02 07:45:02.788713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:02.788656    4515 scope.go:122] "RemoveContainer" containerID="c1ed295ae538458eedee1f4f6d483a0ad4936f8d76945e001e388b0c6a659b31"
Nov 02 07:45:03.361428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:03.361398    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:45:03.362626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:03.362592    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:45:03.363563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:03.363541    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:45:03.817195 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:03.817147    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerStarted","Data":"9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"}
Nov 02 07:45:03.832172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:03.832060    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerStarted","Data":"0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"}
Nov 02 07:45:04.841183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.841135    4515 generic.go:356] "Generic (PLEG): container finished" podID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerID="9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855" exitCode=60
Nov 02 07:45:04.841676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.841218    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerDied","Data":"9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"}
Nov 02 07:45:04.841676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.841343    4515 scope.go:122] "RemoveContainer" containerID="eb4751e0ebda44e5bbd74e321f8b78435cae6b9541510639fd935b874431f294"
Nov 02 07:45:04.841676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.841604    4515 scope.go:122] "RemoveContainer" containerID="9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"
Nov 02 07:45:04.842058 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:04.842035    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893425-g9jq2_odh-ai-gateway-infra(1d27ce22-1af0-418e-b5e6-1e19a0615143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143"
Nov 02 07:45:04.843606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.843586    4515 generic.go:356] "Generic (PLEG): container finished" podID="4f45e798-178a-45a3-b779-29548b7afae8" containerID="0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f" exitCode=60
Nov 02 07:45:04.843724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.843658    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerDied","Data":"0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"}
Nov 02 07:45:04.844035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.844020    4515 scope.go:122] "RemoveContainer" containerID="0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"
Nov 02 07:45:04.844399 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:04.844379    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6_odh-ai-gateway-infra(4f45e798-178a-45a3-b779-29548b7afae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" podUID="4f45e798-178a-45a3-b779-29548b7afae8"
Nov 02 07:45:04.854496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:04.854470    4515 scope.go:122] "RemoveContainer" containerID="c1ed295ae538458eedee1f4f6d483a0ad4936f8d76945e001e388b0c6a659b31"
Nov 02 07:45:06.042898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:06.042869    4515 scope.go:122] "RemoveContainer" containerID="9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"
Nov 02 07:45:06.044753 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:06.044697    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893425-g9jq2_odh-ai-gateway-infra(1d27ce22-1af0-418e-b5e6-1e19a0615143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143"
Nov 02 07:45:06.045385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:06.044957    4515 scope.go:122] "RemoveContainer" containerID="0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"
Nov 02 07:45:06.045907 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:06.045883    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6_odh-ai-gateway-infra(4f45e798-178a-45a3-b779-29548b7afae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" podUID="4f45e798-178a-45a3-b779-29548b7afae8"
Nov 02 07:45:06.743601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:06.739457    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:45:06.743601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:06.739537    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:45:07.969767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:07.969729    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:45:08.008386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:08.008352    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:45:08.490405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:08.488653    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:45:11.100344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.099886    4515 generic.go:356] "Generic (PLEG): container finished" podID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerID="c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447" exitCode=0
Nov 02 07:45:11.100344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.100121    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerDied","Data":"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"}
Nov 02 07:45:11.102058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.101412    4515 scope.go:122] "RemoveContainer" containerID="c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"
Nov 02 07:45:11.103643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.103608    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:11.103803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.103775    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:11.103897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:11.103859    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:12.109436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:12.109392    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerStarted","Data":"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"}
Nov 02 07:45:12.572942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:12.572903    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:45:12.573158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:12.573132    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:12.573216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:12.573194    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:12.971107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:12.970955    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" podUID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" containerName="payload-processing" containerID="cri-o://8a42773634f612bf5606325d558ace9ef85679acf9b1adcc28d89be2f59da706" gracePeriod=25
Nov 02 07:45:13.008901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.008863    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" podUID="6994160c-12b1-401a-8acf-9912066465e0" containerName="payload-pre-processing" containerID="cri-o://d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9" gracePeriod=25
Nov 02 07:45:13.150787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.150758    4515 generic.go:356] "Generic (PLEG): container finished" podID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" containerID="8a42773634f612bf5606325d558ace9ef85679acf9b1adcc28d89be2f59da706" exitCode=0
Nov 02 07:45:13.151191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.150852    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" event={"ID":"e623b172-d2ff-44e7-8270-1a2e6a938f3c","Type":"ContainerDied","Data":"8a42773634f612bf5606325d558ace9ef85679acf9b1adcc28d89be2f59da706"}
Nov 02 07:45:13.395891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.395866    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:45:13.455902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.455696    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:45:13.473208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.473157    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\" (UniqueName: \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\") pod \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") "
Nov 02 07:45:13.473415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.473218    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\") pod \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\" (UID: \"e623b172-d2ff-44e7-8270-1a2e6a938f3c\") "
Nov 02 07:45:13.473415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.473294    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\") pod \"6994160c-12b1-401a-8acf-9912066465e0\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") "
Nov 02 07:45:13.473818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.473574    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\" (UniqueName: \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\") pod \"6994160c-12b1-401a-8acf-9912066465e0\" (UID: \"6994160c-12b1-401a-8acf-9912066465e0\") "
Nov 02 07:45:13.474524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.474108    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume" pod "6994160c-12b1-401a-8acf-9912066465e0" (UID: "6994160c-12b1-401a-8acf-9912066465e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:13.474930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.474833    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6994160c-12b1-401a-8acf-9912066465e0-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:13.475192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.475169    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume" pod "e623b172-d2ff-44e7-8270-1a2e6a938f3c" (UID: "e623b172-d2ff-44e7-8270-1a2e6a938f3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:13.476297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.476275    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt" pod "e623b172-d2ff-44e7-8270-1a2e6a938f3c" (UID: "e623b172-d2ff-44e7-8270-1a2e6a938f3c"). InnerVolumeSpecName "kube-api-access-qrwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:13.477521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.477498    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh" pod "6994160c-12b1-401a-8acf-9912066465e0" (UID: "6994160c-12b1-401a-8acf-9912066465e0"). InnerVolumeSpecName "kube-api-access-7bhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:13.572770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.572736    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:13.572982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.572819    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:13.576884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.576859    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhhh\" (UniqueName: \"kubernetes.io/projected/6994160c-12b1-401a-8acf-9912066465e0-kube-api-access-7bhhh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:13.576884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.576885    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwjt\" (UniqueName: \"kubernetes.io/projected/e623b172-d2ff-44e7-8270-1a2e6a938f3c-kube-api-access-qrwjt\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:13.577055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:13.576895    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e623b172-d2ff-44e7-8270-1a2e6a938f3c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:14.159003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.158968    4515 generic.go:356] "Generic (PLEG): container finished" podID="6994160c-12b1-401a-8acf-9912066465e0" containerID="d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9" exitCode=0
Nov 02 07:45:14.159486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.159014    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"
Nov 02 07:45:14.159486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.159051    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" event={"ID":"6994160c-12b1-401a-8acf-9912066465e0","Type":"ContainerDied","Data":"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"}
Nov 02 07:45:14.159486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.159101    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj" event={"ID":"6994160c-12b1-401a-8acf-9912066465e0","Type":"ContainerDied","Data":"e7f60065d1e1f1dee21d0bcba4f60f4204ebd03215e9a85fd97f184a0882cc63"}
Nov 02 07:45:14.159486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.159124    4515 scope.go:122] "RemoveContainer" containerID="d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"
Nov 02 07:45:14.160905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.160882    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"
Nov 02 07:45:14.160905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.160891    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt" event={"ID":"e623b172-d2ff-44e7-8270-1a2e6a938f3c","Type":"ContainerDied","Data":"376bedbf182ed0fb534a098b5939e4f0899c2a2a13314423269e641f9d071acc"}
Nov 02 07:45:14.171629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.171613    4515 scope.go:122] "RemoveContainer" containerID="d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"
Nov 02 07:45:14.171880 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:14.171861    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9\": container with ID starting with d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9 not found: ID does not exist" containerID="d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"
Nov 02 07:45:14.171926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.171888    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9"} err="failed to get container status \"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9\": rpc error: code = NotFound desc = could not find container \"d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9\": container with ID starting with d66cc3bb14463adf10924613f45756ca94785bcf56dac58fbbc88fe45a620ea9 not found: ID does not exist"
Nov 02 07:45:14.171926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.171906    4515 scope.go:122] "RemoveContainer" containerID="8a42773634f612bf5606325d558ace9ef85679acf9b1adcc28d89be2f59da706"
Nov 02 07:45:14.213642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.213603    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:45:14.244426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.244326    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77dc65fb-784fcb597f-n2fvj"]
Nov 02 07:45:14.266003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.265974    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:45:14.269465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.269444    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77dc65fb-785b45cf9-k64dt"]
Nov 02 07:45:14.572840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.572811    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:14.573028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:14.572875    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:15.391337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:15.367351    4515 scope.go:122] "RemoveContainer" containerID="9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"
Nov 02 07:45:15.835336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:15.819275    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:15.835336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:15.819351    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:16.159272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.159208    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6994160c-12b1-401a-8acf-9912066465e0" path="/var/lib/kubelet/pods/6994160c-12b1-401a-8acf-9912066465e0/volumes"
Nov 02 07:45:16.160421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.160390    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" path="/var/lib/kubelet/pods/e623b172-d2ff-44e7-8270-1a2e6a938f3c/volumes"
Nov 02 07:45:16.160855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.160834    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:45:16.162894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.162853    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6994160c-12b1-401a-8acf-9912066465e0" containerName="payload-pre-processing"
Nov 02 07:45:16.162894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.162883    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6994160c-12b1-401a-8acf-9912066465e0" containerName="payload-pre-processing"
Nov 02 07:45:16.163128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.162969    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" containerName="payload-processing"
Nov 02 07:45:16.163128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.162978    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" containerName="payload-processing"
Nov 02 07:45:16.163226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.163157    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e623b172-d2ff-44e7-8270-1a2e6a938f3c" containerName="payload-processing"
Nov 02 07:45:16.163226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.163176    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6994160c-12b1-401a-8acf-9912066465e0" containerName="payload-pre-processing"
Nov 02 07:45:16.169064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.168997    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:45:16.169131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.169109    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.184325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.184290    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6cc3f7\""
Nov 02 07:45:16.196871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.196854    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6cc3f7-dockercfg-wxrpd\""
Nov 02 07:45:16.342547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.342478    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkgf\" (UniqueName: \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.342924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.342885    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.444494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.444394    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.444896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.444542    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkgf\" (UniqueName: \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.445455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.445430    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.468161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.468129    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkgf\" (UniqueName: \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\") pod \"payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.486266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.486234    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:16.647142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.647015    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:16.647142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:16.647076    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:17.007269 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:17.007239    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe10f7b_e0f6_45be_a25f_61b86be2adcf.slice/crio-0b311b0964d5f616753872f03e8a2089ce3b02294993c8176d2704dd9e02bdb5 WatchSource:0}: Error finding container 0b311b0964d5f616753872f03e8a2089ce3b02294993c8176d2704dd9e02bdb5: Status 404 returned error can't find the container with id 0b311b0964d5f616753872f03e8a2089ce3b02294993c8176d2704dd9e02bdb5
Nov 02 07:45:17.008160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.008136    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:45:17.572656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.572615    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 07:45:17.573091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.572691    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 07:45:17.718446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.718409    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" event={"ID":"4fe10f7b-e0f6-45be-a25f-61b86be2adcf","Type":"ContainerStarted","Data":"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"}
Nov 02 07:45:17.718446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.718474    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" event={"ID":"4fe10f7b-e0f6-45be-a25f-61b86be2adcf","Type":"ContainerStarted","Data":"0b311b0964d5f616753872f03e8a2089ce3b02294993c8176d2704dd9e02bdb5"}
Nov 02 07:45:17.718803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.718518    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:17.721718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.721688    4515 generic.go:356] "Generic (PLEG): container finished" podID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerID="8cd685c07126cdfb54aeb2e82b26fc18838141cafc8cd162d3d94e1e947a4f52" exitCode=60
Nov 02 07:45:17.721861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.721803    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerDied","Data":"8cd685c07126cdfb54aeb2e82b26fc18838141cafc8cd162d3d94e1e947a4f52"}
Nov 02 07:45:17.721861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.721855    4515 scope.go:122] "RemoveContainer" containerID="9138b7bb4f6d5c3e19ba1c55e74fd93e70dbfd46da235ad9a74662d22cc5f855"
Nov 02 07:45:17.722464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.722390    4515 scope.go:122] "RemoveContainer" containerID="8cd685c07126cdfb54aeb2e82b26fc18838141cafc8cd162d3d94e1e947a4f52"
Nov 02 07:45:17.722814 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:17.722792    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893425-g9jq2_odh-ai-gateway-infra(1d27ce22-1af0-418e-b5e6-1e19a0615143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143"
Nov 02 07:45:17.731117 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:17.731090    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:45:17.762524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:17.762469    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" podStartSLOduration=2.762454864 podStartE2EDuration="2.762454864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:17.761379459 +0000 UTC m=+1262.523816366" watchObservedRunningTime="2026-11-02 07:45:17.762454864 +0000 UTC m=+1262.524891769"
Nov 02 07:45:18.366420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:18.366393    4515 scope.go:122] "RemoveContainer" containerID="0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"
Nov 02 07:45:18.577081 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:18.577049    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:45:18.577596 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:18.577557    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:45:18.578464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:18.578440    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:45:18.825135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:18.825106    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:19.095709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.095685    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:19.125182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.125148    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\") pod \"1d27ce22-1af0-418e-b5e6-1e19a0615143\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") "
Nov 02 07:45:19.125329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.125247    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\" (UniqueName: \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\") pod \"1d27ce22-1af0-418e-b5e6-1e19a0615143\" (UID: \"1d27ce22-1af0-418e-b5e6-1e19a0615143\") "
Nov 02 07:45:19.125515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.125494    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca" pod "1d27ce22-1af0-418e-b5e6-1e19a0615143" (UID: "1d27ce22-1af0-418e-b5e6-1e19a0615143"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:19.125913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.125897    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d27ce22-1af0-418e-b5e6-1e19a0615143-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.127581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.127547    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns" pod "1d27ce22-1af0-418e-b5e6-1e19a0615143" (UID: "1d27ce22-1af0-418e-b5e6-1e19a0615143"). InnerVolumeSpecName "kube-api-access-ddqns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:19.227534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.227495    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddqns\" (UniqueName: \"kubernetes.io/projected/1d27ce22-1af0-418e-b5e6-1e19a0615143-kube-api-access-ddqns\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.745721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.745636    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"
Nov 02 07:45:19.745721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.745673    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2" event={"ID":"1d27ce22-1af0-418e-b5e6-1e19a0615143","Type":"ContainerDied","Data":"6edcd9df19aa64fc4f38f7d53bdf58a1bebf8559cec268204de0cc5c1d5e0d5d"}
Nov 02 07:45:19.746234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.745746    4515 scope.go:122] "RemoveContainer" containerID="8cd685c07126cdfb54aeb2e82b26fc18838141cafc8cd162d3d94e1e947a4f52"
Nov 02 07:45:19.747934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.747907    4515 generic.go:356] "Generic (PLEG): container finished" podID="4f45e798-178a-45a3-b779-29548b7afae8" containerID="45fc872eed687872c0aa23b7dd0d12eb17764a091658429a7a19b462e126ec8d" exitCode=60
Nov 02 07:45:19.748207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.748004    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerDied","Data":"45fc872eed687872c0aa23b7dd0d12eb17764a091658429a7a19b462e126ec8d"}
Nov 02 07:45:19.748680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.748664    4515 scope.go:122] "RemoveContainer" containerID="45fc872eed687872c0aa23b7dd0d12eb17764a091658429a7a19b462e126ec8d"
Nov 02 07:45:19.748969 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:19.748951    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6_odh-ai-gateway-infra(4f45e798-178a-45a3-b779-29548b7afae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" podUID="4f45e798-178a-45a3-b779-29548b7afae8"
Nov 02 07:45:19.756486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.756464    4515 scope.go:122] "RemoveContainer" containerID="0ab53379ae76f1d125f4fe27ee66e74268ba206242453e4c0926adeeb1c47b1f"
Nov 02 07:45:19.802232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.802196    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:19.811034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:19.811001    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893425-g9jq2"]
Nov 02 07:45:20.857278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:20.857244    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:21.031660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.031630    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:21.114702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.114619    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\") pod \"4f45e798-178a-45a3-b779-29548b7afae8\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") "
Nov 02 07:45:21.114869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.114710    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\" (UniqueName: \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\") pod \"4f45e798-178a-45a3-b779-29548b7afae8\" (UID: \"4f45e798-178a-45a3-b779-29548b7afae8\") "
Nov 02 07:45:21.115025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.115001    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca" pod "4f45e798-178a-45a3-b779-29548b7afae8" (UID: "4f45e798-178a-45a3-b779-29548b7afae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.115396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.115377    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f45e798-178a-45a3-b779-29548b7afae8-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.116759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.116730    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz" pod "4f45e798-178a-45a3-b779-29548b7afae8" (UID: "4f45e798-178a-45a3-b779-29548b7afae8"). InnerVolumeSpecName "kube-api-access-jvwhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.217748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.217711    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwhz\" (UniqueName: \"kubernetes.io/projected/4f45e798-178a-45a3-b779-29548b7afae8-kube-api-access-jvwhz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.375249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.375173    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" path="/var/lib/kubelet/pods/1d27ce22-1af0-418e-b5e6-1e19a0615143/volumes"
Nov 02 07:45:21.768889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.768863    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"
Nov 02 07:45:21.769044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.768898    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6" event={"ID":"4f45e798-178a-45a3-b779-29548b7afae8","Type":"ContainerDied","Data":"eb2e4a2be3e860c6db09b3e0116d69accf8161242cfc577bde16cf17851eeab7"}
Nov 02 07:45:21.769044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.768958    4515 scope.go:122] "RemoveContainer" containerID="45fc872eed687872c0aa23b7dd0d12eb17764a091658429a7a19b462e126ec8d"
Nov 02 07:45:21.826120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.826084    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:21.836734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:21.836704    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d12c2-29893425-sq4g6"]
Nov 02 07:45:23.376142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:23.376102    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f45e798-178a-45a3-b779-29548b7afae8" path="/var/lib/kubelet/pods/4f45e798-178a-45a3-b779-29548b7afae8/volumes"
Nov 02 07:45:26.998412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:26.998374    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:45:27.000035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000004    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.000205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000193    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.000356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000345    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.000451 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000441    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.000537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000528    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.000599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000591    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.000685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000676    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.000745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000736    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.000831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000821    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.000895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.000886    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.001112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.001099    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.001228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.001217    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.008401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.008376    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.008606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.008594    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.008712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.008702    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.022226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.022178    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.039590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.039555    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:45:27.059816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.059787    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d12c2-dockercfg-gnb75\""
Nov 02 07:45:27.060086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.060070    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d12c2\""
Nov 02 07:45:27.141572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.141517    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.142115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.142078    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbk5\" (UniqueName: \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.166953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.166424    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:45:27.173256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.167686    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.173256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.167742    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d27ce22-1af0-418e-b5e6-1e19a0615143" containerName="cleanup"
Nov 02 07:45:27.173256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.167966    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f45e798-178a-45a3-b779-29548b7afae8" containerName="cleanup"
Nov 02 07:45:27.176508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.176472    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.209887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.209838    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:45:27.245241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.245201    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.250349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.250297    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbk5\" (UniqueName: \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.251244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.246469    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.274770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.274743    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbk5\" (UniqueName: \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\") pod \"payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.353028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.352987    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvtk\" (UniqueName: \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.353217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.353127    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.417678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.417632    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.455060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.455024    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvtk\" (UniqueName: \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.455258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.455231    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.456087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.456062    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.480783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.480752    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvtk\" (UniqueName: \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.517793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.517198    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:27.733350 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:27.733273    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b2540a_0639_45bf_b49c_c1def6cda2b7.slice/crio-d67c7bc98dafff7858dd56b60a10f703f4192edc903321e59905ff06d345375c WatchSource:0}: Error finding container d67c7bc98dafff7858dd56b60a10f703f4192edc903321e59905ff06d345375c: Status 404 returned error can't find the container with id d67c7bc98dafff7858dd56b60a10f703f4192edc903321e59905ff06d345375c
Nov 02 07:45:27.733767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.733739    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:45:27.868667 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:27.868639    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd504ff8a_614f_4241_8403_9453fd12189c.slice/crio-c3b31d5d646acbb94be080ee9e5f3694a83f573c24dd63605cb4f5cfae8727d5 WatchSource:0}: Error finding container c3b31d5d646acbb94be080ee9e5f3694a83f573c24dd63605cb4f5cfae8727d5: Status 404 returned error can't find the container with id c3b31d5d646acbb94be080ee9e5f3694a83f573c24dd63605cb4f5cfae8727d5
Nov 02 07:45:27.871097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.871066    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:45:27.936686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.936654    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" event={"ID":"b9b2540a-0639-45bf-b49c-c1def6cda2b7","Type":"ContainerStarted","Data":"0e3983e3bc7b8e1f6ccc00cb3c8d896085d76a73eb4a01c39af22bbf9296ba56"}
Nov 02 07:45:27.936838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.936716    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" event={"ID":"b9b2540a-0639-45bf-b49c-c1def6cda2b7","Type":"ContainerStarted","Data":"d67c7bc98dafff7858dd56b60a10f703f4192edc903321e59905ff06d345375c"}
Nov 02 07:45:27.936838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.936747    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:27.938825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:27.938792    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" event={"ID":"d504ff8a-614f-4241-8403-9453fd12189c","Type":"ContainerStarted","Data":"c3b31d5d646acbb94be080ee9e5f3694a83f573c24dd63605cb4f5cfae8727d5"}
Nov 02 07:45:28.051445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:28.036669    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" podStartSLOduration=2.036646739 podStartE2EDuration="2.036646739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:28.026244657 +0000 UTC m=+1272.788681566" watchObservedRunningTime="2026-11-02 07:45:28.036646739 +0000 UTC m=+1272.799083667"
Nov 02 07:45:28.735065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:28.735034    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:45:28.949019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:28.948978    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" event={"ID":"d504ff8a-614f-4241-8403-9453fd12189c","Type":"ContainerStarted","Data":"75725f6e96f243e1c9c8a313af7df9e9c51c84812c464b75cfd10096b44232a4"}
Nov 02 07:45:28.949250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:28.949202    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:28.993242 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:28.993115    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" podStartSLOduration=1.993101594 podStartE2EDuration="1.993101594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:28.991165578 +0000 UTC m=+1273.753602485" watchObservedRunningTime="2026-11-02 07:45:28.993101594 +0000 UTC m=+1273.755538500"
Nov 02 07:45:30.481735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.481702    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:45:30.486415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.486392    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.503836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.503812    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-54fd3c\""
Nov 02 07:45:30.505740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.505712    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:45:30.513970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.513938    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-54fd3c-dockercfg-zq9jz\""
Nov 02 07:45:30.545887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.545845    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8q6\" (UniqueName: \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.546084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.545993    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.684333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.680851    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.684333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.681243    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8q6\" (UniqueName: \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.684333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.682525    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.763391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.749048    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8q6\" (UniqueName: \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\") pod \"payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:30.808376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:30.806401    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:31.261089 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:31.261050    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec1cfa4_0d0f_4acc_b993_291281fd96fd.slice/crio-9d6cdb82cc49e8bee5ade50df20b805310dee94c54919cb8929bd8c930b49dcf WatchSource:0}: Error finding container 9d6cdb82cc49e8bee5ade50df20b805310dee94c54919cb8929bd8c930b49dcf: Status 404 returned error can't find the container with id 9d6cdb82cc49e8bee5ade50df20b805310dee94c54919cb8929bd8c930b49dcf
Nov 02 07:45:31.261566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:31.261545    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:45:31.985995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:31.985953    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" event={"ID":"4ec1cfa4-0d0f-4acc-b993-291281fd96fd","Type":"ContainerStarted","Data":"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"}
Nov 02 07:45:31.986551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:31.986068    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:31.986551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:31.986098    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" event={"ID":"4ec1cfa4-0d0f-4acc-b993-291281fd96fd","Type":"ContainerStarted","Data":"9d6cdb82cc49e8bee5ade50df20b805310dee94c54919cb8929bd8c930b49dcf"}
Nov 02 07:45:32.030393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.030344    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" podStartSLOduration=2.030330922 podStartE2EDuration="2.030330922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:32.027489673 +0000 UTC m=+1276.789926579" watchObservedRunningTime="2026-11-02 07:45:32.030330922 +0000 UTC m=+1276.792767822"
Nov 02 07:45:32.410493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.410457    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:45:32.414766 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.414750    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.434570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.434542    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2cc1f\""
Nov 02 07:45:32.435223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.435203    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2cc1f-dockercfg-pq7nk\""
Nov 02 07:45:32.435999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.435965    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:45:32.559661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.556954    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.559661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.557089    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cfz\" (UniqueName: \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.755837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.735639    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cfz\" (UniqueName: \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.755837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.736569    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.755837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.738645    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:32.875333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:32.856116    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cfz\" (UniqueName: \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:33.031194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:33.031093    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:33.560304 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:45:33.560231    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85b78e7_ea39_4bbd_ac27_22fc47daf865.slice/crio-f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732 WatchSource:0}: Error finding container f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732: Status 404 returned error can't find the container with id f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732
Nov 02 07:45:33.560733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:33.560712    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:45:34.001764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:34.001724    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" event={"ID":"f85b78e7-ea39-4bbd-ac27-22fc47daf865","Type":"ContainerStarted","Data":"61648711f1e167b9a5f096c2f453bdfd02d8319994d20772ebd27dcb38716665"}
Nov 02 07:45:34.001975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:34.001793    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:34.001975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:34.001811    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" event={"ID":"f85b78e7-ea39-4bbd-ac27-22fc47daf865","Type":"ContainerStarted","Data":"f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732"}
Nov 02 07:45:34.053122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:34.053072    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" podStartSLOduration=2.053056206 podStartE2EDuration="2.053056206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:34.05089964 +0000 UTC m=+1278.813336546" watchObservedRunningTime="2026-11-02 07:45:34.053056206 +0000 UTC m=+1278.815493143"
Nov 02 07:45:37.672330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:37.670216    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:45:38.964334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:38.963524    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:45:39.957858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:39.957827    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:45:42.997209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:42.997169    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:45:45.153253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:45:45.153219    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:45:47.724507 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:45:47.724468    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:46:05.320246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:05.320162    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:46:05.320857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:05.320409    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy" containerID="cri-o://275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e" gracePeriod=30
Nov 02 07:46:05.333373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:05.333330    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:46:05.333603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:05.333579    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy" containerID="cri-o://8c2d38a8bdf6c523ba649aa6e7b355c15a0d2f8fe164547acf916d434cf2b752" gracePeriod=30
Nov 02 07:46:06.358498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.358464    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:46:06.358977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.358723    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" podUID="98f44f08-ce08-4d84-a906-d6de952d6e2c" containerName="maas-api" containerID="cri-o://96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927" gracePeriod=30
Nov 02 07:46:06.564056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.562459    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:46:06.564056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.562766    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" podUID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" containerName="payload-pre-processing" containerID="cri-o://2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a" gracePeriod=30
Nov 02 07:46:06.708811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.708780    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:46:06.841230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.841200    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:46:06.940384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.931762    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\" (UniqueName: \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\") pod \"98f44f08-ce08-4d84-a906-d6de952d6e2c\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") "
Nov 02 07:46:06.940384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.931940    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\") pod \"98f44f08-ce08-4d84-a906-d6de952d6e2c\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") "
Nov 02 07:46:06.940384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.932043    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\") pod \"98f44f08-ce08-4d84-a906-d6de952d6e2c\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") "
Nov 02 07:46:06.940384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.932322    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\") pod \"98f44f08-ce08-4d84-a906-d6de952d6e2c\" (UID: \"98f44f08-ce08-4d84-a906-d6de952d6e2c\") "
Nov 02 07:46:06.941043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.940957    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca" pod "98f44f08-ce08-4d84-a906-d6de952d6e2c" (UID: "98f44f08-ce08-4d84-a906-d6de952d6e2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:06.994333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.974597    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd" pod "98f44f08-ce08-4d84-a906-d6de952d6e2c" (UID: "98f44f08-ce08-4d84-a906-d6de952d6e2c"). InnerVolumeSpecName "kube-api-access-hzlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:06.994333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:06.982444    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs" pod "98f44f08-ce08-4d84-a906-d6de952d6e2c" (UID: "98f44f08-ce08-4d84-a906-d6de952d6e2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:07.004339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.001328    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls" pod "98f44f08-ce08-4d84-a906-d6de952d6e2c" (UID: "98f44f08-ce08-4d84-a906-d6de952d6e2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:07.042333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.034830    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") pod \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") "
Nov 02 07:46:07.042333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.035292    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\") pod \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") "
Nov 02 07:46:07.042333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.040535    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume" pod "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" (UID: "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:07.093333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.079866    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f44f08-ce08-4d84-a906-d6de952d6e2c-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.093333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.079909    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzlpd\" (UniqueName: \"kubernetes.io/projected/98f44f08-ce08-4d84-a906-d6de952d6e2c-kube-api-access-hzlpd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.093333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.079925    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.093333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.079938    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f44f08-ce08-4d84-a906-d6de952d6e2c-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.193160 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.193067    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq" pod "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" (UID: "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2"). InnerVolumeSpecName "kube-api-access-gbczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:07.194839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.194814    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") pod \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\" (UID: \"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2\") "
Nov 02 07:46:07.195264 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:07.195088    4515 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2/volumes/kubernetes.io~projected/kube-api-access-gbczq
Nov 02 07:46:07.195264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.195225    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq" pod "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" (UID: "2bbdfa55-acd6-4821-a64b-bc209ff6e5d2"). InnerVolumeSpecName "kube-api-access-gbczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:07.196233 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.196209    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.196388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.196372    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbczq\" (UniqueName: \"kubernetes.io/projected/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2-kube-api-access-gbczq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.482355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.482109    4515 generic.go:356] "Generic (PLEG): container finished" podID="98f44f08-ce08-4d84-a906-d6de952d6e2c" containerID="96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927" exitCode=0
Nov 02 07:46:07.482355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.482190    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"
Nov 02 07:46:07.482355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.482201    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" event={"ID":"98f44f08-ce08-4d84-a906-d6de952d6e2c","Type":"ContainerDied","Data":"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"}
Nov 02 07:46:07.482355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.482267    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b" event={"ID":"98f44f08-ce08-4d84-a906-d6de952d6e2c","Type":"ContainerDied","Data":"4058f86c4a3d610603f6f48d71f6a46c7194f5153706ed324a6b5742085359fc"}
Nov 02 07:46:07.482355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.482298    4515 scope.go:122] "RemoveContainer" containerID="96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"
Nov 02 07:46:07.484474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.484451    4515 generic.go:356] "Generic (PLEG): container finished" podID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" containerID="2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a" exitCode=0
Nov 02 07:46:07.484588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.484499    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"
Nov 02 07:46:07.484588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.484542    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" event={"ID":"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2","Type":"ContainerDied","Data":"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"}
Nov 02 07:46:07.484682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.484593    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq" event={"ID":"2bbdfa55-acd6-4821-a64b-bc209ff6e5d2","Type":"ContainerDied","Data":"554b8901f79fbdd8b09cc4fe9664e0ea470393ae3f66f39223bde0bdb2b1db19"}
Nov 02 07:46:07.495391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.495372    4515 scope.go:122] "RemoveContainer" containerID="96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"
Nov 02 07:46:07.495776 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:07.495725    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927\": container with ID starting with 96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927 not found: ID does not exist" containerID="96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"
Nov 02 07:46:07.495900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.495766    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927"} err="failed to get container status \"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927\": rpc error: code = NotFound desc = could not find container \"96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927\": container with ID starting with 96112d61af05922711c46f0202ff51d7bcea2e26065912d9bea6ad7b77e65927 not found: ID does not exist"
Nov 02 07:46:07.495900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.495792    4515 scope.go:122] "RemoveContainer" containerID="2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"
Nov 02 07:46:07.506173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.506156    4515 scope.go:122] "RemoveContainer" containerID="2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"
Nov 02 07:46:07.506635 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:07.506618    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a\": container with ID starting with 2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a not found: ID does not exist" containerID="2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"
Nov 02 07:46:07.506706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.506643    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a"} err="failed to get container status \"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a\": rpc error: code = NotFound desc = could not find container \"2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a\": container with ID starting with 2855ba8ce3977028e646218482d673b594a676f9f88f25a2be5e805291d24c5a not found: ID does not exist"
Nov 02 07:46:07.553135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.553104    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:46:07.565551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.565514    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e07c6-7895765595-vgg9b"]
Nov 02 07:46:07.605607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.605576    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:46:07.609211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:07.609180    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e07c6-79cff67d6c-glkfq"]
Nov 02 07:46:09.384606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:09.384570    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" path="/var/lib/kubelet/pods/2bbdfa55-acd6-4821-a64b-bc209ff6e5d2/volumes"
Nov 02 07:46:09.385321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:09.385286    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f44f08-ce08-4d84-a906-d6de952d6e2c" path="/var/lib/kubelet/pods/98f44f08-ce08-4d84-a906-d6de952d6e2c/volumes"
Nov 02 07:46:10.237369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.237334    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:46:10.237648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.237619    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy" containerID="cri-o://cc672a5e27e2aef22fa00be2384d8177b03b1a385fc8a12c385e9c1d5f7c6b47" gracePeriod=30
Nov 02 07:46:10.581368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.576052    4515 generic.go:356] "Generic (PLEG): container finished" podID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerID="8c2d38a8bdf6c523ba649aa6e7b355c15a0d2f8fe164547acf916d434cf2b752" exitCode=0
Nov 02 07:46:10.581368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.579501    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" event={"ID":"be82cb21-73ce-4648-a095-cb44cd4b22ee","Type":"ContainerDied","Data":"8c2d38a8bdf6c523ba649aa6e7b355c15a0d2f8fe164547acf916d434cf2b752"}
Nov 02 07:46:10.818485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.818416    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:46:10.849225 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.849189    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:46:10.938605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.938570    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.938800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.938659    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.938955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.938905    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.938955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.938909    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.938955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.938943    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939028    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939081    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939159    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939193    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939227    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939276    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939395    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939440    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939553    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939603    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939680    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939706    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\") pod \"c23baa86-b864-4223-adc5-f8a7124cbe70\" (UID: \"c23baa86-b864-4223-adc5-f8a7124cbe70\") "
Nov 02 07:46:10.939851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939830    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.939905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.939874    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\") pod \"be82cb21-73ce-4648-a095-cb44cd4b22ee\" (UID: \"be82cb21-73ce-4648-a095-cb44cd4b22ee\") "
Nov 02 07:46:10.940707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.940681    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:10.940807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.940711    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:10.941674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.941642    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.941783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.941719    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:10.941863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.941842    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.941937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.941916    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.942166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.942142    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:10.943438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.943233    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "kube-api-access-pjn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:10.943612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.943584    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:10.943753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.943618    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.944061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.944042    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.944383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.944360    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.944522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.944435    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.945547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.944808    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "kube-api-access-qpknf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:10.945547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.945230    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:10.946664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.946639    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo" pod "c23baa86-b864-4223-adc5-f8a7124cbe70" (UID: "c23baa86-b864-4223-adc5-f8a7124cbe70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:10.946948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.946920    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:10.948365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:10.948301    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo" pod "be82cb21-73ce-4648-a095-cb44cd4b22ee" (UID: "be82cb21-73ce-4648-a095-cb44cd4b22ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053319    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053354    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053365    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053375    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053383    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82cb21-73ce-4648-a095-cb44cd4b22ee-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053392    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053400    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053408    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82cb21-73ce-4648-a095-cb44cd4b22ee-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053418    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c23baa86-b864-4223-adc5-f8a7124cbe70-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053432    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053440    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpknf\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-kube-api-access-qpknf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053450    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053458    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c23baa86-b864-4223-adc5-f8a7124cbe70-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053467    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82cb21-73ce-4648-a095-cb44cd4b22ee-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053475    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjn89\" (UniqueName: \"kubernetes.io/projected/be82cb21-73ce-4648-a095-cb44cd4b22ee-kube-api-access-pjn89\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.053505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.053483    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c23baa86-b864-4223-adc5-f8a7124cbe70-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:11.589173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.589139    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"
Nov 02 07:46:11.589721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.589176    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz" event={"ID":"be82cb21-73ce-4648-a095-cb44cd4b22ee","Type":"ContainerDied","Data":"0dba2fb97035992fac51c54980b3fa67fbfb2d4d244b83345222c36f773d0289"}
Nov 02 07:46:11.589721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.589249    4515 scope.go:122] "RemoveContainer" containerID="8c2d38a8bdf6c523ba649aa6e7b355c15a0d2f8fe164547acf916d434cf2b752"
Nov 02 07:46:11.591261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.591236    4515 generic.go:356] "Generic (PLEG): container finished" podID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerID="275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e" exitCode=0
Nov 02 07:46:11.591415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.591273    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" event={"ID":"c23baa86-b864-4223-adc5-f8a7124cbe70","Type":"ContainerDied","Data":"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"}
Nov 02 07:46:11.591415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.591296    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"
Nov 02 07:46:11.591525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.591334    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj" event={"ID":"c23baa86-b864-4223-adc5-f8a7124cbe70","Type":"ContainerDied","Data":"f75c08f16d80d753aeb2806f3370e435773cb98d82a68e1c53cb0b7205a236f3"}
Nov 02 07:46:11.602923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.602901    4515 scope.go:122] "RemoveContainer" containerID="275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"
Nov 02 07:46:11.613025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.612990    4515 scope.go:122] "RemoveContainer" containerID="275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"
Nov 02 07:46:11.613380 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:11.613359    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e\": container with ID starting with 275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e not found: ID does not exist" containerID="275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"
Nov 02 07:46:11.613466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.613387    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e"} err="failed to get container status \"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e\": rpc error: code = NotFound desc = could not find container \"275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e\": container with ID starting with 275afca6fa7cad96e2f74e7b81608ac0b89cb12539c9ce172affc1a933d5b04e not found: ID does not exist"
Nov 02 07:46:11.644871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.644786    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:46:11.657623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.657590    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-688jz"]
Nov 02 07:46:11.688619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.688588    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:46:11.698686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:11.698658    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-9zbxj"]
Nov 02 07:46:13.375445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:13.375413    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" path="/var/lib/kubelet/pods/be82cb21-73ce-4648-a095-cb44cd4b22ee/volumes"
Nov 02 07:46:13.375933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:13.375919    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" path="/var/lib/kubelet/pods/c23baa86-b864-4223-adc5-f8a7124cbe70/volumes"
Nov 02 07:46:15.625500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.625469    4515 generic.go:356] "Generic (PLEG): container finished" podID="962061a8-44c3-4e04-8467-9eeab4761cea" containerID="cc672a5e27e2aef22fa00be2384d8177b03b1a385fc8a12c385e9c1d5f7c6b47" exitCode=0
Nov 02 07:46:15.625942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.625513    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerDied","Data":"cc672a5e27e2aef22fa00be2384d8177b03b1a385fc8a12c385e9c1d5f7c6b47"}
Nov 02 07:46:15.625942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.625564    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm" event={"ID":"962061a8-44c3-4e04-8467-9eeab4761cea","Type":"ContainerDied","Data":"271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73"}
Nov 02 07:46:15.625942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.625580    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271f79695f3bbc65ef122018ca653e9b5bdc648523b9b9771700aeeb58617e73"
Nov 02 07:46:15.625942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.625594    4515 scope.go:122] "RemoveContainer" containerID="50add28e2b3afcad546bbe170590b1bcc8d78be9f1b99f9f19bae6f4a9741538"
Nov 02 07:46:15.645882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.645858    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.753796    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.753890    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.753940    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.753976    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.754073    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.754172    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.754345    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.754370    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.755658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.754456    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\") pod \"962061a8-44c3-4e04-8467-9eeab4761cea\" (UID: \"962061a8-44c3-4e04-8467-9eeab4761cea\") "
Nov 02 07:46:15.759693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.757477    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:15.760963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.760619    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:15.760963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.760925    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:15.763846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.763821    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:15.775493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.775450    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:15.782993 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.782932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:15.803380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.802551    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:15.805335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.804545    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "kube-api-access-4np2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:15.834760 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.834559    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy" pod "962061a8-44c3-4e04-8467-9eeab4761cea" (UID: "962061a8-44c3-4e04-8467-9eeab4761cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:15.857717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857672    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857716    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857730    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857746    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962061a8-44c3-4e04-8467-9eeab4761cea-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857760    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857773    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962061a8-44c3-4e04-8467-9eeab4761cea-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857786    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857799    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4np2t\" (UniqueName: \"kubernetes.io/projected/962061a8-44c3-4e04-8467-9eeab4761cea-kube-api-access-4np2t\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.857813    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962061a8-44c3-4e04-8467-9eeab4761cea-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:15.879424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:15.863987    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:46:16.671391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:16.668210    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"
Nov 02 07:46:16.829421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:16.829377    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:46:16.845980 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:16.845919    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e07c6-openshift-default-59b56678c9-f6dfm"]
Nov 02 07:46:17.374420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:17.374390    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" path="/var/lib/kubelet/pods/962061a8-44c3-4e04-8467-9eeab4761cea/volumes"
Nov 02 07:46:17.795196 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:17.795159    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:46:29.635349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:29.635302    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:46:31.633138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.633103    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:46:31.672909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.672875    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:46:31.789901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.787589    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:46:31.789901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.787671    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:46:31.789901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.787872    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy" containerID="cri-o://bbe778ed9004f986fa297378f12c702f362da5d512a50447c0ddde3b8c6213dd" gracePeriod=30
Nov 02 07:46:31.789901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.788575    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy" containerID="cri-o://e820af28d57fe22038cf1c8fae7da82d03200b2a7d53b4f25f266b733c6e0675" gracePeriod=30
Nov 02 07:46:31.802391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.800088    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:46:31.802391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.800465    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy" containerID="cri-o://3fc1af314b6fd4bd0995f8528e9f898534f34ac1c2ee8824af314b064b4ad7bf" gracePeriod=30
Nov 02 07:46:31.805440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.804959    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:46:31.805440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:31.805179    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy" containerID="cri-o://e2dd26835f35475bbcc115b46bc9a073d471b28225e189be119a006d44f5cb3d" gracePeriod=30
Nov 02 07:46:34.645043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.644997    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" podUID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" containerName="payload-pre-processing" containerID="cri-o://61648711f1e167b9a5f096c2f453bdfd02d8319994d20772ebd27dcb38716665" gracePeriod=25
Nov 02 07:46:34.932559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.932523    4515 generic.go:356] "Generic (PLEG): container finished" podID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" containerID="61648711f1e167b9a5f096c2f453bdfd02d8319994d20772ebd27dcb38716665" exitCode=0
Nov 02 07:46:34.932717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.932697    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" event={"ID":"f85b78e7-ea39-4bbd-ac27-22fc47daf865","Type":"ContainerDied","Data":"61648711f1e167b9a5f096c2f453bdfd02d8319994d20772ebd27dcb38716665"}
Nov 02 07:46:34.932804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.932788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj" event={"ID":"f85b78e7-ea39-4bbd-ac27-22fc47daf865","Type":"ContainerDied","Data":"f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732"}
Nov 02 07:46:34.932866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.932816    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55002901044b8e66a80d21413dd8b227e96a47b1a8a115a20c17ef5c811b732"
Nov 02 07:46:34.949534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:34.949507    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:46:35.078889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.078851    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\" (UniqueName: \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\") pod \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") "
Nov 02 07:46:35.079062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.078907    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\") pod \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\" (UID: \"f85b78e7-ea39-4bbd-ac27-22fc47daf865\") "
Nov 02 07:46:35.079277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.079248    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume" pod "f85b78e7-ea39-4bbd-ac27-22fc47daf865" (UID: "f85b78e7-ea39-4bbd-ac27-22fc47daf865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:35.079590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.079572    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85b78e7-ea39-4bbd-ac27-22fc47daf865-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:35.081097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.081078    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz" pod "f85b78e7-ea39-4bbd-ac27-22fc47daf865" (UID: "f85b78e7-ea39-4bbd-ac27-22fc47daf865"). InnerVolumeSpecName "kube-api-access-q4cfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:35.181203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.181113    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4cfz\" (UniqueName: \"kubernetes.io/projected/f85b78e7-ea39-4bbd-ac27-22fc47daf865-kube-api-access-q4cfz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:35.939679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.939648    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"
Nov 02 07:46:35.995972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:35.995934    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:46:36.006246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.006216    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-77795b444c-5p7vj"]
Nov 02 07:46:36.633914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.633878    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" podUID="32885759-fe9f-4abd-a145-80c4642b0779" containerName="payload-processing" containerID="cri-o://c66fa8dca4a19f70c7221fea9d1d808df6b2c55d61f2d63af00da693954d5b65" gracePeriod=25
Nov 02 07:46:36.673296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.673259    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" podUID="fc8dd368-31dc-4423-8ae8-274b09399d04" containerName="payload-pre-processing" containerID="cri-o://5bbd9160c78dbacf635f4b1a5be251998bc6cafca0964825758b4290b93a8f04" gracePeriod=25
Nov 02 07:46:36.969590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.952328    4515 generic.go:356] "Generic (PLEG): container finished" podID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerID="bbe778ed9004f986fa297378f12c702f362da5d512a50447c0ddde3b8c6213dd" exitCode=0
Nov 02 07:46:36.969590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.952461    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" event={"ID":"7080f30d-98e2-4748-9c7a-c8ba02dd715c","Type":"ContainerDied","Data":"bbe778ed9004f986fa297378f12c702f362da5d512a50447c0ddde3b8c6213dd"}
Nov 02 07:46:36.969590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.961708    4515 generic.go:356] "Generic (PLEG): container finished" podID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerID="e820af28d57fe22038cf1c8fae7da82d03200b2a7d53b4f25f266b733c6e0675" exitCode=0
Nov 02 07:46:36.969590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.961810    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" event={"ID":"33b98d21-89e3-4b52-838b-11e3f86ac7a3","Type":"ContainerDied","Data":"e820af28d57fe22038cf1c8fae7da82d03200b2a7d53b4f25f266b733c6e0675"}
Nov 02 07:46:36.983261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.983042    4515 generic.go:356] "Generic (PLEG): container finished" podID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerID="e2dd26835f35475bbcc115b46bc9a073d471b28225e189be119a006d44f5cb3d" exitCode=0
Nov 02 07:46:36.983261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:36.983146    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" event={"ID":"7d6d9467-bca4-4fe6-9880-688f88c66871","Type":"ContainerDied","Data":"e2dd26835f35475bbcc115b46bc9a073d471b28225e189be119a006d44f5cb3d"}
Nov 02 07:46:37.004945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.004908    4515 generic.go:356] "Generic (PLEG): container finished" podID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerID="3fc1af314b6fd4bd0995f8528e9f898534f34ac1c2ee8824af314b064b4ad7bf" exitCode=0
Nov 02 07:46:37.005121 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.005017    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" event={"ID":"0aafb588-87d9-49ef-9f5d-1c8d928a1db2","Type":"ContainerDied","Data":"3fc1af314b6fd4bd0995f8528e9f898534f34ac1c2ee8824af314b064b4ad7bf"}
Nov 02 07:46:37.025450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.025058    4515 generic.go:356] "Generic (PLEG): container finished" podID="32885759-fe9f-4abd-a145-80c4642b0779" containerID="c66fa8dca4a19f70c7221fea9d1d808df6b2c55d61f2d63af00da693954d5b65" exitCode=0
Nov 02 07:46:37.025450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.025149    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" event={"ID":"32885759-fe9f-4abd-a145-80c4642b0779","Type":"ContainerDied","Data":"c66fa8dca4a19f70c7221fea9d1d808df6b2c55d61f2d63af00da693954d5b65"}
Nov 02 07:46:37.029105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.028896    4515 generic.go:356] "Generic (PLEG): container finished" podID="fc8dd368-31dc-4423-8ae8-274b09399d04" containerID="5bbd9160c78dbacf635f4b1a5be251998bc6cafca0964825758b4290b93a8f04" exitCode=0
Nov 02 07:46:37.029105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.029021    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" event={"ID":"fc8dd368-31dc-4423-8ae8-274b09399d04","Type":"ContainerDied","Data":"5bbd9160c78dbacf635f4b1a5be251998bc6cafca0964825758b4290b93a8f04"}
Nov 02 07:46:37.107662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.107622    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:46:37.132134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.132107    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\" (UniqueName: \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\") pod \"32885759-fe9f-4abd-a145-80c4642b0779\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") "
Nov 02 07:46:37.132345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.132328    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\") pod \"32885759-fe9f-4abd-a145-80c4642b0779\" (UID: \"32885759-fe9f-4abd-a145-80c4642b0779\") "
Nov 02 07:46:37.132740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.132715    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume" pod "32885759-fe9f-4abd-a145-80c4642b0779" (UID: "32885759-fe9f-4abd-a145-80c4642b0779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.132908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.132890    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32885759-fe9f-4abd-a145-80c4642b0779-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.134791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.134769    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww" pod "32885759-fe9f-4abd-a145-80c4642b0779" (UID: "32885759-fe9f-4abd-a145-80c4642b0779"). InnerVolumeSpecName "kube-api-access-tfmww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.199393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.194379    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:46:37.235445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.234259    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\" (UniqueName: \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\") pod \"fc8dd368-31dc-4423-8ae8-274b09399d04\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") "
Nov 02 07:46:37.235445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.234558    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\") pod \"fc8dd368-31dc-4423-8ae8-274b09399d04\" (UID: \"fc8dd368-31dc-4423-8ae8-274b09399d04\") "
Nov 02 07:46:37.235445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.235064    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfmww\" (UniqueName: \"kubernetes.io/projected/32885759-fe9f-4abd-a145-80c4642b0779-kube-api-access-tfmww\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.235445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.235399    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume" pod "fc8dd368-31dc-4423-8ae8-274b09399d04" (UID: "fc8dd368-31dc-4423-8ae8-274b09399d04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.263346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.263270    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq" pod "fc8dd368-31dc-4423-8ae8-274b09399d04" (UID: "fc8dd368-31dc-4423-8ae8-274b09399d04"). InnerVolumeSpecName "kube-api-access-fpsrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.278359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.278293    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:46:37.288153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.286656    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:46:37.307457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.307287    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:46:37.312530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.312505    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:46:37.335923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.335884    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.336105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.335933    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.336105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.335980    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.336105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336028    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.336105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336059    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.336105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336086    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.336469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336113    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.336469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336170    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.336469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336195    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.336469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336209    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.336469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336269    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.336727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336680    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.336960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.337048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.336962    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.337109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337059    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.337171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337118    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.337566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337538    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.337674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337587    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.337743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337686    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.337743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337726    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.337851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337776    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.337851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337813    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.337851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337840    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.338535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.337883    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.338535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338265    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.338535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338361    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.338535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338393    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.338535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338463    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.338851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338061    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.338851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338371    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.338851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338618    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.339010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.338905    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.339371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.339151    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.339371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.339247    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.339371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.339292    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\") pod \"7d6d9467-bca4-4fe6-9880-688f88c66871\" (UID: \"7d6d9467-bca4-4fe6-9880-688f88c66871\") "
Nov 02 07:46:37.340066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340037    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.340134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340095    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.340179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340138    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.340230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340176    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.340230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340218    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.340290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340268    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.340351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340299    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\") pod \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\" (UID: \"33b98d21-89e3-4b52-838b-11e3f86ac7a3\") "
Nov 02 07:46:37.340388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340362    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.340419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340389    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.340452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340439    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\") pod \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\" (UID: \"7080f30d-98e2-4748-9c7a-c8ba02dd715c\") "
Nov 02 07:46:37.340496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.340473    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\") pod \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\" (UID: \"0aafb588-87d9-49ef-9f5d-1c8d928a1db2\") "
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341428    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341455    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341471    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpsrq\" (UniqueName: \"kubernetes.io/projected/fc8dd368-31dc-4423-8ae8-274b09399d04-kube-api-access-fpsrq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341486    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341502    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341516    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8dd368-31dc-4423-8ae8-274b09399d04-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341530    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341543    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341557    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341571    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341586    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.341878 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.341599    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6d9467-bca4-4fe6-9880-688f88c66871-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.342968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.342942    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.343293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.343181    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.343888    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344127    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344148    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "kube-api-access-vllnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344397    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344405    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344642    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.344952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344698    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:37.345529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.344969    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.345529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.345146    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.345529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.345457    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.345790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.345766    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "kube-api-access-sh4gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.347995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.347970    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.348221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.348196    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.348890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.348865    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.348982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.348965    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.350051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350024    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "kube-api-access-jmc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.350595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350514    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:37.350595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350572    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo" pod "7d6d9467-bca4-4fe6-9880-688f88c66871" (UID: "7d6d9467-bca4-4fe6-9880-688f88c66871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:37.350595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350572    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.350804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350623    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.351020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.350944    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:37.351626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.351599    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo" pod "7080f30d-98e2-4748-9c7a-c8ba02dd715c" (UID: "7080f30d-98e2-4748-9c7a-c8ba02dd715c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:37.351832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.351807    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4" pod "0aafb588-87d9-49ef-9f5d-1c8d928a1db2" (UID: "0aafb588-87d9-49ef-9f5d-1c8d928a1db2"). InnerVolumeSpecName "kube-api-access-jh2t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.352156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.352127    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token" pod "33b98d21-89e3-4b52-838b-11e3f86ac7a3" (UID: "33b98d21-89e3-4b52-838b-11e3f86ac7a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:37.380240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.380195    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" path="/var/lib/kubelet/pods/f85b78e7-ea39-4bbd-ac27-22fc47daf865/volumes"
Nov 02 07:46:37.444601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.443116    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444675    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444707    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444722    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444737    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444753    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444767    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmc9t\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-kube-api-access-jmc9t\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444783    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2t4\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-kube-api-access-jh2t4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444798    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b98d21-89e3-4b52-838b-11e3f86ac7a3-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444812    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444825    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b98d21-89e3-4b52-838b-11e3f86ac7a3-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444839    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444852    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444866    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444878    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444891    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444905    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllnn\" (UniqueName: \"kubernetes.io/projected/33b98d21-89e3-4b52-838b-11e3f86ac7a3-kube-api-access-vllnn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.444928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444918    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444934    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444946    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444959    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444972    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4gs\" (UniqueName: \"kubernetes.io/projected/7d6d9467-bca4-4fe6-9880-688f88c66871-kube-api-access-sh4gs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444987    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080f30d-98e2-4748-9c7a-c8ba02dd715c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.444999    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.445010    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6d9467-bca4-4fe6-9880-688f88c66871-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:37.445303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:37.445022    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aafb588-87d9-49ef-9f5d-1c8d928a1db2-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:38.042398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.042345    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv" event={"ID":"33b98d21-89e3-4b52-838b-11e3f86ac7a3","Type":"ContainerDied","Data":"99b819b4926f73d0568691fe441ac1b203c0eaceb650beafed2911799889620f"}
Nov 02 07:46:38.042832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.042405    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"
Nov 02 07:46:38.042832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.042473    4515 scope.go:122] "RemoveContainer" containerID="e820af28d57fe22038cf1c8fae7da82d03200b2a7d53b4f25f266b733c6e0675"
Nov 02 07:46:38.044828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.044801    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"
Nov 02 07:46:38.045552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.044800    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst" event={"ID":"7d6d9467-bca4-4fe6-9880-688f88c66871","Type":"ContainerDied","Data":"5eb85f2f736837c18f59018a5db2fd12bc38e744703db5a258028494e936adaa"}
Nov 02 07:46:38.046758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.046737    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"
Nov 02 07:46:38.046885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.046738    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2" event={"ID":"0aafb588-87d9-49ef-9f5d-1c8d928a1db2","Type":"ContainerDied","Data":"4ef9e16bc88f6e389463bfa801f5e80f5ae9c913866401fac1b8cb440ac3060c"}
Nov 02 07:46:38.048213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.048193    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"
Nov 02 07:46:38.048331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.048200    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf" event={"ID":"32885759-fe9f-4abd-a145-80c4642b0779","Type":"ContainerDied","Data":"3eacc404104c0a5ec63707c4095027a96bbb37395efe2661764bdaa4aafff289"}
Nov 02 07:46:38.049846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.049831    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"
Nov 02 07:46:38.049940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.049923    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt" event={"ID":"fc8dd368-31dc-4423-8ae8-274b09399d04","Type":"ContainerDied","Data":"d084cddb8e4b5266f3b9aa728f31d0d9c75ab9758221f8487c541ad39e36fe48"}
Nov 02 07:46:38.051551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.051529    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb" event={"ID":"7080f30d-98e2-4748-9c7a-c8ba02dd715c","Type":"ContainerDied","Data":"75cddb65d1ec01748390cf1b724000d7e63fab19e896e277beb0a8569011bc16"}
Nov 02 07:46:38.051642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.051598    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"
Nov 02 07:46:38.055061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.055044    4515 scope.go:122] "RemoveContainer" containerID="e2dd26835f35475bbcc115b46bc9a073d471b28225e189be119a006d44f5cb3d"
Nov 02 07:46:38.101974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.101674    4515 scope.go:122] "RemoveContainer" containerID="3fc1af314b6fd4bd0995f8528e9f898534f34ac1c2ee8824af314b064b4ad7bf"
Nov 02 07:46:38.131351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.130644    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:46:38.135226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.135191    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5a032f-5f87f8ff7-nfkgf"]
Nov 02 07:46:38.140609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.139929    4515 scope.go:122] "RemoveContainer" containerID="c66fa8dca4a19f70c7221fea9d1d808df6b2c55d61f2d63af00da693954d5b65"
Nov 02 07:46:38.153615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.153594    4515 scope.go:122] "RemoveContainer" containerID="5bbd9160c78dbacf635f4b1a5be251998bc6cafca0964825758b4290b93a8f04"
Nov 02 07:46:38.163416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.163395    4515 scope.go:122] "RemoveContainer" containerID="bbe778ed9004f986fa297378f12c702f362da5d512a50447c0ddde3b8c6213dd"
Nov 02 07:46:38.168132 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.168110    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:46:38.174303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.174281    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8thl2"]
Nov 02 07:46:38.208159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.208131    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:46:38.214735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.214710    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5a032f-75cd84b75f-5hxjt"]
Nov 02 07:46:38.249989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.249960    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:46:38.253265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.253239    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-6wtst"]
Nov 02 07:46:38.288370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.288207    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:46:38.293791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.293728    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-5sbrb"]
Nov 02 07:46:38.328945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.328906    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:46:38.333908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.333883    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-dq6fv"]
Nov 02 07:46:38.339569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.339545    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:46:38.339784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.339760    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" podUID="5a751671-2b34-4f20-b010-9d907ef59693" containerName="maas-api" containerID="cri-o://d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada" gracePeriod=30
Nov 02 07:46:38.703482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.703454    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:46:38.776294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.775055    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\") pod \"5a751671-2b34-4f20-b010-9d907ef59693\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") "
Nov 02 07:46:38.776294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.775157    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\") pod \"5a751671-2b34-4f20-b010-9d907ef59693\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") "
Nov 02 07:46:38.776294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.775234    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\" (UniqueName: \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\") pod \"5a751671-2b34-4f20-b010-9d907ef59693\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") "
Nov 02 07:46:38.776294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.775267    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\") pod \"5a751671-2b34-4f20-b010-9d907ef59693\" (UID: \"5a751671-2b34-4f20-b010-9d907ef59693\") "
Nov 02 07:46:38.796331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.795577    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca" pod "5a751671-2b34-4f20-b010-9d907ef59693" (UID: "5a751671-2b34-4f20-b010-9d907ef59693"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:38.811334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.808290    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs" pod "5a751671-2b34-4f20-b010-9d907ef59693" (UID: "5a751671-2b34-4f20-b010-9d907ef59693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:38.822333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.816622    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm" pod "5a751671-2b34-4f20-b010-9d907ef59693" (UID: "5a751671-2b34-4f20-b010-9d907ef59693"). InnerVolumeSpecName "kube-api-access-dv8zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:38.856385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.852292    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls" pod "5a751671-2b34-4f20-b010-9d907ef59693" (UID: "5a751671-2b34-4f20-b010-9d907ef59693"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:38.893337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.892491    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:38.893337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.892533    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a751671-2b34-4f20-b010-9d907ef59693-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:38.893337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.892573    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8zm\" (UniqueName: \"kubernetes.io/projected/5a751671-2b34-4f20-b010-9d907ef59693-kube-api-access-dv8zm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:38.893337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:38.892648    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a751671-2b34-4f20-b010-9d907ef59693-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:39.138385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.132942    4515 generic.go:356] "Generic (PLEG): container finished" podID="5a751671-2b34-4f20-b010-9d907ef59693" containerID="d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada" exitCode=0
Nov 02 07:46:39.138385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.133167    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" event={"ID":"5a751671-2b34-4f20-b010-9d907ef59693","Type":"ContainerDied","Data":"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"}
Nov 02 07:46:39.138385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.133218    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" event={"ID":"5a751671-2b34-4f20-b010-9d907ef59693","Type":"ContainerDied","Data":"f6e9dde8e2317452c765c1fdd142eba74db58d34e8a39fd77d83432d147eae0e"}
Nov 02 07:46:39.138385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.133241    4515 scope.go:122] "RemoveContainer" containerID="d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"
Nov 02 07:46:39.138385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.133338    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"
Nov 02 07:46:39.167729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.167708    4515 scope.go:122] "RemoveContainer" containerID="d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"
Nov 02 07:46:39.168807 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:39.168265    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada\": container with ID starting with d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada not found: ID does not exist" containerID="d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"
Nov 02 07:46:39.169019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.168300    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada"} err="failed to get container status \"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada\": rpc error: code = NotFound desc = could not find container \"d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada\": container with ID starting with d5b102117111767cc83f5d3d8cb5f5a403788dff0eb8c6e1e34eeb0de6ba0ada not found: ID does not exist"
Nov 02 07:46:39.189793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.189757    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:46:39.202080 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.202050    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb"]
Nov 02 07:46:39.375259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.375226    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" path="/var/lib/kubelet/pods/0aafb588-87d9-49ef-9f5d-1c8d928a1db2/volumes"
Nov 02 07:46:39.375782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.375763    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32885759-fe9f-4abd-a145-80c4642b0779" path="/var/lib/kubelet/pods/32885759-fe9f-4abd-a145-80c4642b0779/volumes"
Nov 02 07:46:39.376244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.376230    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" path="/var/lib/kubelet/pods/33b98d21-89e3-4b52-838b-11e3f86ac7a3/volumes"
Nov 02 07:46:39.377030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.377014    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a751671-2b34-4f20-b010-9d907ef59693" path="/var/lib/kubelet/pods/5a751671-2b34-4f20-b010-9d907ef59693/volumes"
Nov 02 07:46:39.377473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.377459    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" path="/var/lib/kubelet/pods/7080f30d-98e2-4748-9c7a-c8ba02dd715c/volumes"
Nov 02 07:46:39.377891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.377878    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" path="/var/lib/kubelet/pods/7d6d9467-bca4-4fe6-9880-688f88c66871/volumes"
Nov 02 07:46:39.378603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:39.378587    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8dd368-31dc-4423-8ae8-274b09399d04" path="/var/lib/kubelet/pods/fc8dd368-31dc-4423-8ae8-274b09399d04/volumes"
Nov 02 07:46:41.601472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:41.601430    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5a032f-746485bd6f-vdcmb" podUID="5a751671-2b34-4f20-b010-9d907ef59693" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.114:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:42.707128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.707092    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:46:42.707620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.707343    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" podUID="59b90158-0fe7-4b33-b562-d566bb916c17" containerName="maas-api" containerID="cri-o://1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f" gracePeriod=30
Nov 02 07:46:42.872519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.872472    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:46:42.872747 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.872723    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" podUID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" containerName="payload-processing" containerID="cri-o://0e3983e3bc7b8e1f6ccc00cb3c8d896085d76a73eb4a01c39af22bbf9296ba56" gracePeriod=30
Nov 02 07:46:42.902532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.902494    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:46:42.903250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:42.903021    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" podUID="d504ff8a-614f-4241-8403-9453fd12189c" containerName="payload-pre-processing" containerID="cri-o://75725f6e96f243e1c9c8a313af7df9e9c51c84812c464b75cfd10096b44232a4" gracePeriod=30
Nov 02 07:46:43.055279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.055250    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:46:43.180736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.180490    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\") pod \"59b90158-0fe7-4b33-b562-d566bb916c17\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") "
Nov 02 07:46:43.180736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.180546    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\") pod \"59b90158-0fe7-4b33-b562-d566bb916c17\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") "
Nov 02 07:46:43.180736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.180732    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\") pod \"59b90158-0fe7-4b33-b562-d566bb916c17\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") "
Nov 02 07:46:43.180922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.180800    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\" (UniqueName: \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\") pod \"59b90158-0fe7-4b33-b562-d566bb916c17\" (UID: \"59b90158-0fe7-4b33-b562-d566bb916c17\") "
Nov 02 07:46:43.185789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.181980    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca" pod "59b90158-0fe7-4b33-b562-d566bb916c17" (UID: "59b90158-0fe7-4b33-b562-d566bb916c17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:43.185789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.185527    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls" pod "59b90158-0fe7-4b33-b562-d566bb916c17" (UID: "59b90158-0fe7-4b33-b562-d566bb916c17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:43.185789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.185582    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk" pod "59b90158-0fe7-4b33-b562-d566bb916c17" (UID: "59b90158-0fe7-4b33-b562-d566bb916c17"). InnerVolumeSpecName "kube-api-access-vzmwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:43.186928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.186895    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs" pod "59b90158-0fe7-4b33-b562-d566bb916c17" (UID: "59b90158-0fe7-4b33-b562-d566bb916c17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:46:43.187575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.187544    4515 generic.go:356] "Generic (PLEG): container finished" podID="59b90158-0fe7-4b33-b562-d566bb916c17" containerID="1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f" exitCode=0
Nov 02 07:46:43.187698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.187589    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"
Nov 02 07:46:43.187757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.187725    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" event={"ID":"59b90158-0fe7-4b33-b562-d566bb916c17","Type":"ContainerDied","Data":"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"}
Nov 02 07:46:43.187813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.187776    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj" event={"ID":"59b90158-0fe7-4b33-b562-d566bb916c17","Type":"ContainerDied","Data":"03745984781c5645e574a5605041dcf8008130a21bb4c5f6a95330254cdc0750"}
Nov 02 07:46:43.187860 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.187810    4515 scope.go:122] "RemoveContainer" containerID="1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"
Nov 02 07:46:43.191533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.190670    4515 generic.go:356] "Generic (PLEG): container finished" podID="d504ff8a-614f-4241-8403-9453fd12189c" containerID="75725f6e96f243e1c9c8a313af7df9e9c51c84812c464b75cfd10096b44232a4" exitCode=0
Nov 02 07:46:43.191533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.190814    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" event={"ID":"d504ff8a-614f-4241-8403-9453fd12189c","Type":"ContainerDied","Data":"75725f6e96f243e1c9c8a313af7df9e9c51c84812c464b75cfd10096b44232a4"}
Nov 02 07:46:43.193168 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.193136    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:46:43.194108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.194080    4515 generic.go:356] "Generic (PLEG): container finished" podID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" containerID="0e3983e3bc7b8e1f6ccc00cb3c8d896085d76a73eb4a01c39af22bbf9296ba56" exitCode=0
Nov 02 07:46:43.194221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.194128    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" event={"ID":"b9b2540a-0639-45bf-b49c-c1def6cda2b7","Type":"ContainerDied","Data":"0e3983e3bc7b8e1f6ccc00cb3c8d896085d76a73eb4a01c39af22bbf9296ba56"}
Nov 02 07:46:43.205338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.205290    4515 scope.go:122] "RemoveContainer" containerID="1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"
Nov 02 07:46:43.205671 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:43.205638    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f\": container with ID starting with 1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f not found: ID does not exist" containerID="1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"
Nov 02 07:46:43.205759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.205678    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f"} err="failed to get container status \"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f\": rpc error: code = NotFound desc = could not find container \"1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f\": container with ID starting with 1474135af165dfa4703773e470878600b6dcd02bcb0b8a1290d1a25c27b0154f not found: ID does not exist"
Nov 02 07:46:43.205759 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.205695    4515 scope.go:122] "RemoveContainer" containerID="0e3983e3bc7b8e1f6ccc00cb3c8d896085d76a73eb4a01c39af22bbf9296ba56"
Nov 02 07:46:43.334334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.313943    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.334334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.313975    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmwk\" (UniqueName: \"kubernetes.io/projected/59b90158-0fe7-4b33-b562-d566bb916c17-kube-api-access-vzmwk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.334334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.313990    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b90158-0fe7-4b33-b562-d566bb916c17-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.334334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.314004    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b90158-0fe7-4b33-b562-d566bb916c17-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.436334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.427647    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:46:43.436334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.427698    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d12c2-7ccf7d9ccf-fg5tj"]
Nov 02 07:46:43.465337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.455702    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\" (UniqueName: \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\") pod \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") "
Nov 02 07:46:43.465337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.455856    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\") pod \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\" (UID: \"b9b2540a-0639-45bf-b49c-c1def6cda2b7\") "
Nov 02 07:46:43.465337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.458788    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume" pod "b9b2540a-0639-45bf-b49c-c1def6cda2b7" (UID: "b9b2540a-0639-45bf-b49c-c1def6cda2b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:43.488374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.478364    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5" pod "b9b2540a-0639-45bf-b49c-c1def6cda2b7" (UID: "b9b2540a-0639-45bf-b49c-c1def6cda2b7"). InnerVolumeSpecName "kube-api-access-rdbk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:43.508335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.507182    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:46:43.558596 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.558565    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\") pod \"d504ff8a-614f-4241-8403-9453fd12189c\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") "
Nov 02 07:46:43.558771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.558708    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\" (UniqueName: \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\") pod \"d504ff8a-614f-4241-8403-9453fd12189c\" (UID: \"d504ff8a-614f-4241-8403-9453fd12189c\") "
Nov 02 07:46:43.558983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.558957    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume" pod "d504ff8a-614f-4241-8403-9453fd12189c" (UID: "d504ff8a-614f-4241-8403-9453fd12189c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:43.559258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.559242    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504ff8a-614f-4241-8403-9453fd12189c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.559301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.559260    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbk5\" (UniqueName: \"kubernetes.io/projected/b9b2540a-0639-45bf-b49c-c1def6cda2b7-kube-api-access-rdbk5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.559301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.559269    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b2540a-0639-45bf-b49c-c1def6cda2b7-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:43.561599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.561574    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk" pod "d504ff8a-614f-4241-8403-9453fd12189c" (UID: "d504ff8a-614f-4241-8403-9453fd12189c"). InnerVolumeSpecName "kube-api-access-vpvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:43.660476 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:43.660396    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvtk\" (UniqueName: \"kubernetes.io/projected/d504ff8a-614f-4241-8403-9453fd12189c-kube-api-access-vpvtk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:44.201241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.201204    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"
Nov 02 07:46:44.201755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.201259    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4" event={"ID":"b9b2540a-0639-45bf-b49c-c1def6cda2b7","Type":"ContainerDied","Data":"d67c7bc98dafff7858dd56b60a10f703f4192edc903321e59905ff06d345375c"}
Nov 02 07:46:44.203831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.203806    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv" event={"ID":"d504ff8a-614f-4241-8403-9453fd12189c","Type":"ContainerDied","Data":"c3b31d5d646acbb94be080ee9e5f3694a83f573c24dd63605cb4f5cfae8727d5"}
Nov 02 07:46:44.203968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.203850    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"
Nov 02 07:46:44.203968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.203855    4515 scope.go:122] "RemoveContainer" containerID="75725f6e96f243e1c9c8a313af7df9e9c51c84812c464b75cfd10096b44232a4"
Nov 02 07:46:44.255992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.255961    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:46:44.259281 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.259253    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-746c854599-lqwjv"]
Nov 02 07:46:44.294861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.294825    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:46:44.299091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:44.299061    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-67bcb7fc86-7svt4"]
Nov 02 07:46:45.374668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.374634    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b90158-0fe7-4b33-b562-d566bb916c17" path="/var/lib/kubelet/pods/59b90158-0fe7-4b33-b562-d566bb916c17/volumes"
Nov 02 07:46:45.375077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.375062    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" path="/var/lib/kubelet/pods/b9b2540a-0639-45bf-b49c-c1def6cda2b7/volumes"
Nov 02 07:46:45.375460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.375446    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d504ff8a-614f-4241-8403-9453fd12189c" path="/var/lib/kubelet/pods/d504ff8a-614f-4241-8403-9453fd12189c/volumes"
Nov 02 07:46:45.582172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.582122    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:46:45.582421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.582399    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy" containerID="cri-o://752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e" gracePeriod=30
Nov 02 07:46:45.594684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.594641    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:46:45.598498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.598468    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:46:45.598661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.598493    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy" containerID="cri-o://015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4" gracePeriod=30
Nov 02 07:46:45.598661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:45.598635    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy" containerID="cri-o://c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813" gracePeriod=30
Nov 02 07:46:47.811331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.808619    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816202    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5a751671-2b34-4f20-b010-9d907ef59693" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816234    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a751671-2b34-4f20-b010-9d907ef59693" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816259    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816266    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816284    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816292    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816333    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d504ff8a-614f-4241-8403-9453fd12189c" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816341    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d504ff8a-614f-4241-8403-9453fd12189c" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816353    4515 cpu_manager.go:414] "containerMap: removing container" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816362    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816378    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816386    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816399    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816406    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816418    4515 cpu_manager.go:414] "containerMap: removing container" podUID="98f44f08-ce08-4d84-a906-d6de952d6e2c" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816426    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f44f08-ce08-4d84-a906-d6de952d6e2c" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816439    4515 cpu_manager.go:414] "containerMap: removing container" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816445    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816455    4515 cpu_manager.go:414] "containerMap: removing container" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816462    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816485    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816492    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816503    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816510    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816525    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816571    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816586    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fc8dd368-31dc-4423-8ae8-274b09399d04" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816593    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8dd368-31dc-4423-8ae8-274b09399d04" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816608    4515 cpu_manager.go:414] "containerMap: removing container" podUID="32885759-fe9f-4abd-a145-80c4642b0779" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816616    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32885759-fe9f-4abd-a145-80c4642b0779" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816627    4515 cpu_manager.go:414] "containerMap: removing container" podUID="59b90158-0fe7-4b33-b562-d566bb916c17" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816634    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b90158-0fe7-4b33-b562-d566bb916c17" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816833    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23baa86-b864-4223-adc5-f8a7124cbe70" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816849    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d504ff8a-614f-4241-8403-9453fd12189c" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816858    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="be82cb21-73ce-4648-a095-cb44cd4b22ee" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816871    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85b78e7-ea39-4bbd-ac27-22fc47daf865" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816880    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6d9467-bca4-4fe6-9880-688f88c66871" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816891    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f44f08-ce08-4d84-a906-d6de952d6e2c" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816902    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b98d21-89e3-4b52-838b-11e3f86ac7a3" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816911    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816924    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbdfa55-acd6-4821-a64b-bc209ff6e5d2" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816935    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aafb588-87d9-49ef-9f5d-1c8d928a1db2" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816943    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b2540a-0639-45bf-b49c-c1def6cda2b7" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816954    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816964    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8dd368-31dc-4423-8ae8-274b09399d04" containerName="payload-pre-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816972    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="32885759-fe9f-4abd-a145-80c4642b0779" containerName="payload-processing"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816980    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b90158-0fe7-4b33-b562-d566bb916c17" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.816994    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a751671-2b34-4f20-b010-9d907ef59693" containerName="maas-api"
Nov 02 07:46:47.820524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.817006    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080f30d-98e2-4748-9c7a-c8ba02dd715c" containerName="istio-proxy"
Nov 02 07:46:47.925892 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.920298    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:47.925892 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:47.923097    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:46:47.940452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.928597    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:46:47.949593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.949558    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2cc1f\""
Nov 02 07:46:47.952772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.952746    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2cc1f-dockercfg-7svbq\""
Nov 02 07:46:47.965734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.965704    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:46:47.968867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:47.968532    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" containerID="cri-o://a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0" gracePeriod=30
Nov 02 07:46:48.033511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.033481    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:46:48.034137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.034118    4515 cpu_manager.go:414] "containerMap: removing container" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:48.034137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.034137    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962061a8-44c3-4e04-8467-9eeab4761cea" containerName="istio-proxy"
Nov 02 07:46:48.037749 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.037726    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.047500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.047429    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5ld\" (UniqueName: \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.047855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.047642    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.058544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.058513    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:46:48.164382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.163754    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.164382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.163956    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62ln\" (UniqueName: \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.164382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.164082    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5ld\" (UniqueName: \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.164382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.164166    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.197420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.165220    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.302336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.275693    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62ln\" (UniqueName: \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.302336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.275925    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.302336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.277038    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.310352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.303496    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5ld\" (UniqueName: \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\") pod \"payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.310352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.308783    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62ln\" (UniqueName: \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\") pod \"payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.358459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.358426    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:48.573541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.573504    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:48.579716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.579680    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:46:48.579821 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.579737    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:46:48.671213 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:48.671185    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0daf4db_46d7_4436_bb10_36502f10797e.slice/crio-b2a8ba910e9b846d1b8a1e2c39247ef10d9b67751be01c2023bca7a81fac779c WatchSource:0}: Error finding container b2a8ba910e9b846d1b8a1e2c39247ef10d9b67751be01c2023bca7a81fac779c: Status 404 returned error can't find the container with id b2a8ba910e9b846d1b8a1e2c39247ef10d9b67751be01c2023bca7a81fac779c
Nov 02 07:46:48.673614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.673580    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:46:48.929899 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:48.929831    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd085b0b4_87a5_45c7_ac9e_c627df117592.slice/crio-1822f1b8a05f37d0622845cf489c9e6320bcb804930685ff6c2ed417a9317dde WatchSource:0}: Error finding container 1822f1b8a05f37d0622845cf489c9e6320bcb804930685ff6c2ed417a9317dde: Status 404 returned error can't find the container with id 1822f1b8a05f37d0622845cf489c9e6320bcb804930685ff6c2ed417a9317dde
Nov 02 07:46:48.930814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:48.930733    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:46:49.343185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.343139    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" event={"ID":"d085b0b4-87a5-45c7-ac9e-c627df117592","Type":"ContainerStarted","Data":"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"}
Nov 02 07:46:49.343412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.343235    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:46:49.343412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.343263    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" event={"ID":"d085b0b4-87a5-45c7-ac9e-c627df117592","Type":"ContainerStarted","Data":"1822f1b8a05f37d0622845cf489c9e6320bcb804930685ff6c2ed417a9317dde"}
Nov 02 07:46:49.345074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.345044    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" event={"ID":"a0daf4db-46d7-4436-bb10-36502f10797e","Type":"ContainerStarted","Data":"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"}
Nov 02 07:46:49.345200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.345094    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:46:49.345200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.345111    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" event={"ID":"a0daf4db-46d7-4436-bb10-36502f10797e","Type":"ContainerStarted","Data":"b2a8ba910e9b846d1b8a1e2c39247ef10d9b67751be01c2023bca7a81fac779c"}
Nov 02 07:46:49.387302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.387241    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" podStartSLOduration=2.3872237 podStartE2EDuration="2.3872237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:49.385668744 +0000 UTC m=+1354.148105654" watchObservedRunningTime="2026-11-02 07:46:49.3872237 +0000 UTC m=+1354.149660609"
Nov 02 07:46:49.434441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.434379    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" podStartSLOduration=2.434360819 podStartE2EDuration="2.434360819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:49.429840978 +0000 UTC m=+1354.192277887" watchObservedRunningTime="2026-11-02 07:46:49.434360819 +0000 UTC m=+1354.196797734"
Nov 02 07:46:49.847063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.847031    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:46:49.851397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.851369    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.882984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.882950    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:46:49.898484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.898022    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b7fde8-openshift-default-dockercfg-9srmh\""
Nov 02 07:46:49.899963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.899927    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900091    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900184    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900274    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900366    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900440    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900511    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsnk\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900563    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:49.900677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:49.900583    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002065    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002157    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsnk\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002196    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002210    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002266    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002358    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002396    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002458    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.002620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002483    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.003023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.002691    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.003077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.003037    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.003126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.003078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.003178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.003122    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.003362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.003298    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.005347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.005324    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.005657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.005634    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.061669 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.056326    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsnk\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.065589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.063994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\") pod \"e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") " pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.198385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.187580    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:50.634129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.634100    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:46:50.666220 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:50.666185    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1468920_9ced_4001_a0e4_4721f031c853.slice/crio-0bafe696106df2772fad4b1d3fabf3750bd337411f1e84e349a32a2688653a4d WatchSource:0}: Error finding container 0bafe696106df2772fad4b1d3fabf3750bd337411f1e84e349a32a2688653a4d: Status 404 returned error can't find the container with id 0bafe696106df2772fad4b1d3fabf3750bd337411f1e84e349a32a2688653a4d
Nov 02 07:46:50.706623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.706583    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:50.706750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.706677    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:50.706750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:50.706719    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:51.003123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.002756    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:46:51.085750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.085725    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:46:51.090809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.090783    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136181    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136256    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136283    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136333    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136445    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136464    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136508    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136685    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.136712    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\") pod \"800e102a-4dd1-46df-bfa3-27a60baa8018\" (UID: \"800e102a-4dd1-46df-bfa3-27a60baa8018\") "
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.137948    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.138089    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.138221    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.138352    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.140385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.138795    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.141219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.140494    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:51.141729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.141701    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "kube-api-access-xfjg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.141729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.141722    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.141870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.141786    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token" pod "800e102a-4dd1-46df-bfa3-27a60baa8018" (UID: "800e102a-4dd1-46df-bfa3-27a60baa8018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.238490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238454    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238502    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238544    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238574    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238689    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.238800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238758    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238790    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238840    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238854    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238874    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238925    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.238974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.238927    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239022    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239052    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239071    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239134    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239156    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239192    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239239    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:51.239270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239255    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\") pod \"b349a217-1c19-44b1-a62c-4d25fea8bd10\" (UID: \"b349a217-1c19-44b1-a62c-4d25fea8bd10\") "
Nov 02 07:46:51.239767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239381    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239399    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.239767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239447    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\") pod \"46284b42-5f56-41c0-a1d3-0489a5929b64\" (UID: \"46284b42-5f56-41c0-a1d3-0489a5929b64\") "
Nov 02 07:46:51.239767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239597    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.240037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.239765    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.240149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240122    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240151    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b349a217-1c19-44b1-a62c-4d25fea8bd10-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240166    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240179    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240193    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240206    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240219    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjg5\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-kube-api-access-xfjg5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240233    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240246    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240258    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240272    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240285    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800e102a-4dd1-46df-bfa3-27a60baa8018-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240297    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240329    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240341    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240354    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240366    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800e102a-4dd1-46df-bfa3-27a60baa8018-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.240495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.240380    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800e102a-4dd1-46df-bfa3-27a60baa8018-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.241412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.241387    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.241497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.241411    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "kube-api-access-xdpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.241751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.241727    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:51.242193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.242160    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.242421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.242380    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "kube-api-access-pdcvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.242489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.242424    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:51.243884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.243863    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:51.244401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.244381    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo" pod "b349a217-1c19-44b1-a62c-4d25fea8bd10" (UID: "b349a217-1c19-44b1-a62c-4d25fea8bd10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:51.244939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.244924    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo" pod "46284b42-5f56-41c0-a1d3-0489a5929b64" (UID: "46284b42-5f56-41c0-a1d3-0489a5929b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:51.341620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341585    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46284b42-5f56-41c0-a1d3-0489a5929b64-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341615    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcvn\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-kube-api-access-pdcvn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341626    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341634    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341646    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341657    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341667    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b349a217-1c19-44b1-a62c-4d25fea8bd10-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341679    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpfq\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-kube-api-access-xdpfq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.341918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.341688    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46284b42-5f56-41c0-a1d3-0489a5929b64-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:51.365107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.365076    4515 generic.go:356] "Generic (PLEG): container finished" podID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerID="c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813" exitCode=0
Nov 02 07:46:51.365272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.365124    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"
Nov 02 07:46:51.365272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.365164    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" event={"ID":"b349a217-1c19-44b1-a62c-4d25fea8bd10","Type":"ContainerDied","Data":"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"}
Nov 02 07:46:51.365272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.365250    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2" event={"ID":"b349a217-1c19-44b1-a62c-4d25fea8bd10","Type":"ContainerDied","Data":"d48b0232665a437e2eef731cd02916b37e65e1ab4ca885e401f5020d80b0815b"}
Nov 02 07:46:51.365454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.365330    4515 scope.go:122] "RemoveContainer" containerID="c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"
Nov 02 07:46:51.366837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.366825    4515 generic.go:356] "Generic (PLEG): container finished" podID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerID="015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4" exitCode=0
Nov 02 07:46:51.366962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.366948    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"
Nov 02 07:46:51.368448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.368426    4515 generic.go:356] "Generic (PLEG): container finished" podID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerID="752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e" exitCode=0
Nov 02 07:46:51.368575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.368542    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"
Nov 02 07:46:51.379060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.378879    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" event={"ID":"46284b42-5f56-41c0-a1d3-0489a5929b64","Type":"ContainerDied","Data":"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"}
Nov 02 07:46:51.379060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.378931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p" event={"ID":"46284b42-5f56-41c0-a1d3-0489a5929b64","Type":"ContainerDied","Data":"7dd62936b4a4c082aa8e76cf5c2cc08af3f64178ed09144eee5ab9d4076d26f5"}
Nov 02 07:46:51.379060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.378956    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" event={"ID":"800e102a-4dd1-46df-bfa3-27a60baa8018","Type":"ContainerDied","Data":"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"}
Nov 02 07:46:51.379060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.378983    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v" event={"ID":"800e102a-4dd1-46df-bfa3-27a60baa8018","Type":"ContainerDied","Data":"dae3a9700ebba7494c85842188ffafc4d1b431b5d12c09335074c1a8f1b9002d"}
Nov 02 07:46:51.379060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.379006    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerStarted","Data":"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"}
Nov 02 07:46:51.379373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.379085    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerStarted","Data":"0bafe696106df2772fad4b1d3fabf3750bd337411f1e84e349a32a2688653a4d"}
Nov 02 07:46:51.380854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.380798    4515 scope.go:122] "RemoveContainer" containerID="c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"
Nov 02 07:46:51.381503 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:51.381323    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813\": container with ID starting with c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813 not found: ID does not exist" containerID="c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"
Nov 02 07:46:51.381503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.381357    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813"} err="failed to get container status \"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813\": rpc error: code = NotFound desc = could not find container \"c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813\": container with ID starting with c8ab36671f05d67e5a85ae08992d9434be6d77b60c872c38d4b24d842cac0813 not found: ID does not exist"
Nov 02 07:46:51.381503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.381379    4515 scope.go:122] "RemoveContainer" containerID="015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"
Nov 02 07:46:51.391631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.391613    4515 scope.go:122] "RemoveContainer" containerID="015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"
Nov 02 07:46:51.391915 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:51.391897    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4\": container with ID starting with 015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4 not found: ID does not exist" containerID="015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"
Nov 02 07:46:51.391966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.391925    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4"} err="failed to get container status \"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4\": rpc error: code = NotFound desc = could not find container \"015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4\": container with ID starting with 015c9dd759a8a0c0a2f150a3fd8f786b5dcffeec48cc03d507cf8c0af5a5d0b4 not found: ID does not exist"
Nov 02 07:46:51.391966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.391943    4515 scope.go:122] "RemoveContainer" containerID="752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"
Nov 02 07:46:51.401501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.401480    4515 scope.go:122] "RemoveContainer" containerID="752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"
Nov 02 07:46:51.401745 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:51.401726    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e\": container with ID starting with 752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e not found: ID does not exist" containerID="752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"
Nov 02 07:46:51.401787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.401753    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e"} err="failed to get container status \"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e\": rpc error: code = NotFound desc = could not find container \"752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e\": container with ID starting with 752ec0618b7b16663b80e2506a483ecc12b2a4819876bcfba7defe507a3d1d0e not found: ID does not exist"
Nov 02 07:46:51.427291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.427241    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podStartSLOduration=2.427227107 podStartE2EDuration="2.427227107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:51.426184844 +0000 UTC m=+1356.188621749" watchObservedRunningTime="2026-11-02 07:46:51.427227107 +0000 UTC m=+1356.189664012"
Nov 02 07:46:51.464614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.464586    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:46:51.471896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.471871    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-smwp2"]
Nov 02 07:46:51.564805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.560383    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:46:51.608183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.593018    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-fmr4v"]
Nov 02 07:46:51.626125 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.626073    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:46:51.631656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:51.631625    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lml2p"]
Nov 02 07:46:52.188382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:52.188351    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:52.194237 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:52.194212    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:52.380446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:52.380415    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:52.381414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:52.381391    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:46:52.849798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:52.849764    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:46:53.236200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.236177    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:46:53.282666 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282630    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.282871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282682    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.282871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282715    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.282871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282744    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.282871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282844    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.283106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.282950    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.283106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283003    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.283106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283030    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:53.283106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283044    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.283106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283072    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\") pod \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\" (UID: \"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd\") "
Nov 02 07:46:53.284013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283588    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.284013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283648    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:53.284013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283782    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:53.284013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283812    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:53.284013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.283981    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:53.286171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.286144    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "kube-api-access-lncgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:53.286354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.286147    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:53.287816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.287789    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:53.288012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.287939    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token" pod "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" (UID: "fba2b299-c7d4-4e67-b78e-16cb9f97c7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:53.375275 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.375245    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" path="/var/lib/kubelet/pods/46284b42-5f56-41c0-a1d3-0489a5929b64/volumes"
Nov 02 07:46:53.375783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.375767    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" path="/var/lib/kubelet/pods/800e102a-4dd1-46df-bfa3-27a60baa8018/volumes"
Nov 02 07:46:53.376227 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.376214    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" path="/var/lib/kubelet/pods/b349a217-1c19-44b1-a62c-4d25fea8bd10/volumes"
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385731    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385770    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385785    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385802    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385815    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385833    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385846    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lncgh\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-kube-api-access-lncgh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.387342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.385860    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.399390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.399292    4515 generic.go:356] "Generic (PLEG): container finished" podID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerID="a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0" exitCode=0
Nov 02 07:46:53.399390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.399378    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"
Nov 02 07:46:53.399594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.399379    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerDied","Data":"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"}
Nov 02 07:46:53.399594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.399526    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t" event={"ID":"fba2b299-c7d4-4e67-b78e-16cb9f97c7cd","Type":"ContainerDied","Data":"5ee23306fb50be9161d3454371f9c9625780fd0e96bb21e6519a85503e6b5aad"}
Nov 02 07:46:53.399594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.399555    4515 scope.go:122] "RemoveContainer" containerID="a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"
Nov 02 07:46:53.409489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.409467    4515 scope.go:122] "RemoveContainer" containerID="c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"
Nov 02 07:46:53.418564 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.418542    4515 scope.go:122] "RemoveContainer" containerID="a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"
Nov 02 07:46:53.418856 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:53.418836    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0\": container with ID starting with a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0 not found: ID does not exist" containerID="a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"
Nov 02 07:46:53.418939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.418865    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0"} err="failed to get container status \"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0\": rpc error: code = NotFound desc = could not find container \"a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0\": container with ID starting with a4e29d27c633ee0acd1edd315f062b43a1d6155a5dcc92f8aa1fc646e45b83b0 not found: ID does not exist"
Nov 02 07:46:53.418939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.418883    4515 scope.go:122] "RemoveContainer" containerID="c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"
Nov 02 07:46:53.419199 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:53.419180    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447\": container with ID starting with c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447 not found: ID does not exist" containerID="c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"
Nov 02 07:46:53.419261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.419212    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447"} err="failed to get container status \"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447\": rpc error: code = NotFound desc = could not find container \"c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447\": container with ID starting with c00e3093e3bd50b4fa35c81a91af1eb95409913c82724a6978f70225d5c38447 not found: ID does not exist"
Nov 02 07:46:53.453580 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.453550    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:46:53.462174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.462145    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d12c2-openshift-default-64d7d69647-5vv6t"]
Nov 02 07:46:53.597658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:53.597621    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:46:55.373278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.373243    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" path="/var/lib/kubelet/pods/fba2b299-c7d4-4e67-b78e-16cb9f97c7cd/volumes"
Nov 02 07:46:55.695402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695320    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:46:55.695936 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695923    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695938    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695949    4515 cpu_manager.go:414] "containerMap: removing container" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695954    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695967    4515 cpu_manager.go:414] "containerMap: removing container" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695974    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy"
Nov 02 07:46:55.695987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695986    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy"
Nov 02 07:46:55.696165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.695991    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy"
Nov 02 07:46:55.696165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.696081    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b349a217-1c19-44b1-a62c-4d25fea8bd10" containerName="istio-proxy"
Nov 02 07:46:55.696165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.696091    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:55.696165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.696103    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="800e102a-4dd1-46df-bfa3-27a60baa8018" containerName="istio-proxy"
Nov 02 07:46:55.696165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.696109    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="46284b42-5f56-41c0-a1d3-0489a5929b64" containerName="istio-proxy"
Nov 02 07:46:55.701255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.701234    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:55.719802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.719777    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b7fde8\""
Nov 02 07:46:55.720753 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.720730    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:46:55.729758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.729735    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b7fde8-dockercfg-5m9r2\""
Nov 02 07:46:55.852333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.812159    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnc5w\" (UniqueName: \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:55.883967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.852751    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.988680    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnc5w\" (UniqueName: \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.988850    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.989835    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.990724    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.990741    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.990924    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba2b299-c7d4-4e67-b78e-16cb9f97c7cd" containerName="istio-proxy"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.993637    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.994824    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:46:56.003380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:55.994928    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.030701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.030008    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnc5w\" (UniqueName: \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\") pod \"payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.090194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.090159    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57t22\" (UniqueName: \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.090386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.090224    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.191183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.191143    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57t22\" (UniqueName: \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.191383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.191202    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.192001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.191979    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.214984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.214956    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57t22\" (UniqueName: \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\") pod \"payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.315914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.315826    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:56.320421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.320394    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:56.632211 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:56.632182    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c7d94b_b221_43ef_a22c_baaa92896282.slice/crio-818730a5f1ea88423e9ded14f727af8fe3bfd397a65069d731f0f92a44df166d WatchSource:0}: Error finding container 818730a5f1ea88423e9ded14f727af8fe3bfd397a65069d731f0f92a44df166d: Status 404 returned error can't find the container with id 818730a5f1ea88423e9ded14f727af8fe3bfd397a65069d731f0f92a44df166d
Nov 02 07:46:56.633458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.633430    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:46:56.646067 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:56.646039    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21ba385_c10d_4c41_8093_b8d2910c593e.slice/crio-8c00945c5818667b855535322e87a3f51fd5cd46012b4f1ddeba222a89c0f675 WatchSource:0}: Error finding container 8c00945c5818667b855535322e87a3f51fd5cd46012b4f1ddeba222a89c0f675: Status 404 returned error can't find the container with id 8c00945c5818667b855535322e87a3f51fd5cd46012b4f1ddeba222a89c0f675
Nov 02 07:46:56.646620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:56.646591    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:46:57.463658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.463622    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" event={"ID":"45c7d94b-b221-43ef-a22c-baaa92896282","Type":"ContainerStarted","Data":"41225ca38ab9bc32e7503d4239a082d18d948f92086653c4eadbbb33869dd22e"}
Nov 02 07:46:57.464390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.464357    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" event={"ID":"45c7d94b-b221-43ef-a22c-baaa92896282","Type":"ContainerStarted","Data":"818730a5f1ea88423e9ded14f727af8fe3bfd397a65069d731f0f92a44df166d"}
Nov 02 07:46:57.464637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.464617    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:46:57.465965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.465943    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" event={"ID":"a21ba385-c10d-4c41-8093-b8d2910c593e","Type":"ContainerStarted","Data":"cf8b21952d2a8993a71da8cf28601fb272389ec29510d0990d162aee9072c14d"}
Nov 02 07:46:57.466076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.465983    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" event={"ID":"a21ba385-c10d-4c41-8093-b8d2910c593e","Type":"ContainerStarted","Data":"8c00945c5818667b855535322e87a3f51fd5cd46012b4f1ddeba222a89c0f675"}
Nov 02 07:46:57.466509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.466489    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:46:57.500856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.500818    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:46:57.505593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.505567    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.509827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.509775    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" podStartSLOduration=2.50975795 podStartE2EDuration="2.50975795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:57.506563404 +0000 UTC m=+1362.269000310" watchObservedRunningTime="2026-11-02 07:46:57.50975795 +0000 UTC m=+1362.272194857"
Nov 02 07:46:57.523978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.523945    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dc62c2-dockercfg-2hmk2\""
Nov 02 07:46:57.524134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.524062    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dc62c2\""
Nov 02 07:46:57.529425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.529363    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:46:57.564751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.564710    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.565166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.565131    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7l9\" (UniqueName: \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.594396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.594346    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" podStartSLOduration=2.594331425 podStartE2EDuration="2.594331425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:57.591632793 +0000 UTC m=+1362.354069700" watchObservedRunningTime="2026-11-02 07:46:57.594331425 +0000 UTC m=+1362.356768328"
Nov 02 07:46:57.660479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.660443    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:46:57.664855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.664834    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:57.666178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.666152    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.666383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.666365    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7l9\" (UniqueName: \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.667084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.667061    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.683519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.683491    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:46:57.692964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.692936    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7l9\" (UniqueName: \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\") pod \"payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.830817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.830788    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:57.881167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.881127    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:57.881352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.881188    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847hp\" (UniqueName: \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:57.982941 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.982900    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:57.983111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.982983    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847hp\" (UniqueName: \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:57.983705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:57.983682    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:58.006714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.006683    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847hp\" (UniqueName: \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:58.137727 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:46:58.137697    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440a7324_e2c4_4c54_8bbf_0a4e9b1398ea.slice/crio-d34fdfefdb0ea8f1d0fc1cc274b7be4e74af42c0a7b31e12207120ba2890d853 WatchSource:0}: Error finding container d34fdfefdb0ea8f1d0fc1cc274b7be4e74af42c0a7b31e12207120ba2890d853: Status 404 returned error can't find the container with id d34fdfefdb0ea8f1d0fc1cc274b7be4e74af42c0a7b31e12207120ba2890d853
Nov 02 07:46:58.138632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.138604    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:46:58.182086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.182055    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:46:58.182299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.182274    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy" containerID="cri-o://b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24" gracePeriod=30
Nov 02 07:46:58.185368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.185259    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:46:58.185516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.185486    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy" containerID="cri-o://cb4df52ea4c7c34728528c30aa4d5b95e556b5144beffb6405ce13fc37cb4951" gracePeriod=30
Nov 02 07:46:58.189035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.189006    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:46:58.189262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.189232    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy" containerID="cri-o://78c476639f03add6e6cb05614e3923261c0f9bcc608afd13d0126c5b9a3c17e2" gracePeriod=30
Nov 02 07:46:58.195556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.195530    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:46:58.195919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.195736    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy" containerID="cri-o://26a744d8cb119a4075a7a18a42010c1c8b951d0b60d8c2926665de6384533594" gracePeriod=30
Nov 02 07:46:58.280129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.280088    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:58.483161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.480495    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" event={"ID":"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea","Type":"ContainerStarted","Data":"993f3f7b6da282b2ccca6c3b72f669bf1acd0a7ef6a8f0196fc3dad1c3297ea6"}
Nov 02 07:46:58.483161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.480614    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" event={"ID":"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea","Type":"ContainerStarted","Data":"d34fdfefdb0ea8f1d0fc1cc274b7be4e74af42c0a7b31e12207120ba2890d853"}
Nov 02 07:46:58.539477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.539413    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" podStartSLOduration=1.5393961649999999 podStartE2EDuration="1.539396165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:58.537003733 +0000 UTC m=+1363.299440639" watchObservedRunningTime="2026-11-02 07:46:58.539396165 +0000 UTC m=+1363.301833074"
Nov 02 07:46:58.598398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.598346    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" podUID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" containerName="payload-pre-processing" containerID="cri-o://80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067" gracePeriod=25
Nov 02 07:46:58.742188 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.742144    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:46:58.934945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:58.934916    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:46:59.101645 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.101605    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\" (UniqueName: \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\") pod \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") "
Nov 02 07:46:59.101833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.101674    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\") pod \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\" (UID: \"4ec1cfa4-0d0f-4acc-b993-291281fd96fd\") "
Nov 02 07:46:59.102063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.102039    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume" pod "4ec1cfa4-0d0f-4acc-b993-291281fd96fd" (UID: "4ec1cfa4-0d0f-4acc-b993-291281fd96fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:59.103794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.103771    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6" pod "4ec1cfa4-0d0f-4acc-b993-291281fd96fd" (UID: "4ec1cfa4-0d0f-4acc-b993-291281fd96fd"). InnerVolumeSpecName "kube-api-access-dn8q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:59.203824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.203785    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8q6\" (UniqueName: \"kubernetes.io/projected/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-kube-api-access-dn8q6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:59.203824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.203817    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec1cfa4-0d0f-4acc-b993-291281fd96fd-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:46:59.488663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.488621    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" event={"ID":"1925c757-6df8-4f9f-95c2-650b36fb3296","Type":"ContainerStarted","Data":"d514fb5980c82d47b99910cab0b5e205d8c1c74c01644c397465bd74bd885bb8"}
Nov 02 07:46:59.488836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.488678    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" event={"ID":"1925c757-6df8-4f9f-95c2-650b36fb3296","Type":"ContainerStarted","Data":"864113feb34965efc46ffb721f1a1547f49071c7b8abba79138db21c0aef1de4"}
Nov 02 07:46:59.488836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.488734    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:46:59.490077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490052    4515 generic.go:356] "Generic (PLEG): container finished" podID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" containerID="80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067" exitCode=0
Nov 02 07:46:59.490180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490091    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"
Nov 02 07:46:59.490180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490130    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" event={"ID":"4ec1cfa4-0d0f-4acc-b993-291281fd96fd","Type":"ContainerDied","Data":"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"}
Nov 02 07:46:59.490180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490172    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742" event={"ID":"4ec1cfa4-0d0f-4acc-b993-291281fd96fd","Type":"ContainerDied","Data":"9d6cdb82cc49e8bee5ade50df20b805310dee94c54919cb8929bd8c930b49dcf"}
Nov 02 07:46:59.490343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490198    4515 scope.go:122] "RemoveContainer" containerID="80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"
Nov 02 07:46:59.490662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.490640    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:46:59.499875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.499857    4515 scope.go:122] "RemoveContainer" containerID="80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"
Nov 02 07:46:59.500150 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:46:59.500129    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067\": container with ID starting with 80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067 not found: ID does not exist" containerID="80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"
Nov 02 07:46:59.500236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.500156    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067"} err="failed to get container status \"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067\": rpc error: code = NotFound desc = could not find container \"80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067\": container with ID starting with 80fcf27fbc8bf0ead0e9c8dd5d2c471222b52274fb24cb96cbb8d7bd6ed0c067 not found: ID does not exist"
Nov 02 07:46:59.530920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.530864    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" podStartSLOduration=2.530847532 podStartE2EDuration="2.530847532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:59.529592811 +0000 UTC m=+1364.292029717" watchObservedRunningTime="2026-11-02 07:46:59.530847532 +0000 UTC m=+1364.293284440"
Nov 02 07:46:59.567555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.567522    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:46:59.575433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:46:59.575400    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54fd3c-59c4459b79-5q742"]
Nov 02 07:47:00.358736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:00.358702    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:47:00.359154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:00.358802    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:47:01.373742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:01.373705    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" path="/var/lib/kubelet/pods/4ec1cfa4-0d0f-4acc-b993-291281fd96fd/volumes"
Nov 02 07:47:03.491950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.491922    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:47:03.531979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.531921    4515 generic.go:356] "Generic (PLEG): container finished" podID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerID="b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24" exitCode=0
Nov 02 07:47:03.532159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.531985    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"
Nov 02 07:47:03.532159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.532004    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerDied","Data":"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"}
Nov 02 07:47:03.532159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.532068    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf" event={"ID":"8be33f92-ec9c-447a-ba57-7a2b31ffacc1","Type":"ContainerDied","Data":"73b4976e7c04607eb370d1d7b86c2764fb111731ccebfefe296ed82e33d8884a"}
Nov 02 07:47:03.532159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.532100    4515 scope.go:122] "RemoveContainer" containerID="b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"
Nov 02 07:47:03.536525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.536495    4515 generic.go:356] "Generic (PLEG): container finished" podID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerID="cb4df52ea4c7c34728528c30aa4d5b95e556b5144beffb6405ce13fc37cb4951" exitCode=0
Nov 02 07:47:03.536644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.536557    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" event={"ID":"09be35b7-df0b-45af-a266-609dbbdccdcc","Type":"ContainerDied","Data":"cb4df52ea4c7c34728528c30aa4d5b95e556b5144beffb6405ce13fc37cb4951"}
Nov 02 07:47:03.538219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.538196    4515 generic.go:356] "Generic (PLEG): container finished" podID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerID="78c476639f03add6e6cb05614e3923261c0f9bcc608afd13d0126c5b9a3c17e2" exitCode=0
Nov 02 07:47:03.538409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.538273    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" event={"ID":"c909f7c3-a340-4e1c-b15a-ffe6132e9835","Type":"ContainerDied","Data":"78c476639f03add6e6cb05614e3923261c0f9bcc608afd13d0126c5b9a3c17e2"}
Nov 02 07:47:03.540294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.540270    4515 generic.go:356] "Generic (PLEG): container finished" podID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerID="26a744d8cb119a4075a7a18a42010c1c8b951d0b60d8c2926665de6384533594" exitCode=0
Nov 02 07:47:03.540407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.540356    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" event={"ID":"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c","Type":"ContainerDied","Data":"26a744d8cb119a4075a7a18a42010c1c8b951d0b60d8c2926665de6384533594"}
Nov 02 07:47:03.542904 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.542884    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:47:03.544893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.544877    4515 scope.go:122] "RemoveContainer" containerID="44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"
Nov 02 07:47:03.547522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.547505    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:47:03.554750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.554730    4515 scope.go:122] "RemoveContainer" containerID="b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"
Nov 02 07:47:03.555073 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:03.555045    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24\": container with ID starting with b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24 not found: ID does not exist" containerID="b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"
Nov 02 07:47:03.555164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.555080    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24"} err="failed to get container status \"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24\": rpc error: code = NotFound desc = could not find container \"b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24\": container with ID starting with b7a9f7adc2c9352b949263e49e4258062cd170b3dc478bde51e2fdd413f5eb24 not found: ID does not exist"
Nov 02 07:47:03.555164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.555098    4515 scope.go:122] "RemoveContainer" containerID="44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"
Nov 02 07:47:03.555468 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:03.555439    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7\": container with ID starting with 44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7 not found: ID does not exist" containerID="44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"
Nov 02 07:47:03.555616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.555474    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7"} err="failed to get container status \"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7\": rpc error: code = NotFound desc = could not find container \"44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7\": container with ID starting with 44e1657cfb56b9b879f68126919b288a813bf0c01b7732e67eb9da5c0070a2c7 not found: ID does not exist"
Nov 02 07:47:03.671719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.671687    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.672352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672297    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.672470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672365    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.672470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672391    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.672470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672451    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.672633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672490    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.672633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672525    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.672633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672554    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.672633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672595    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672634    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672639    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672675    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672696    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672714    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672749    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.672822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672796    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672855    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672908    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672973    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.672974    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673012    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673013    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673041    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.673150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673091    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673159    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673222    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673256    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673264    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673336    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673375    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\") pod \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\" (UID: \"8be33f92-ec9c-447a-ba57-7a2b31ffacc1\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673413    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\") pod \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\" (UID: \"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673456    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\") pod \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\" (UID: \"c909f7c3-a340-4e1c-b15a-ffe6132e9835\") "
Nov 02 07:47:03.673657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.673478    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:03.674215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674195    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674222    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674256    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674270    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674301    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674345    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.674611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674571    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.674830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674807    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.674891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674839    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.675047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.674943    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.675047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.675000    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.675576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.675254    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.676469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.676139    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.676469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.676221    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:03.676626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.676471    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.676964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.676943    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.678512 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.678488    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.679739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.679658    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "kube-api-access-s4cfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.679739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.679678    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:03.679739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.679687    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "kube-api-access-wmhb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.680727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.680702    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.680863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.680833    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.680935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.680869    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token" pod "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" (UID: "5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.680999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.680965    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.681060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.681022    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo" pod "c909f7c3-a340-4e1c-b15a-ffe6132e9835" (UID: "c909f7c3-a340-4e1c-b15a-ffe6132e9835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:03.681397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.681378    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.682722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.682699    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654" pod "8be33f92-ec9c-447a-ba57-7a2b31ffacc1" (UID: "8be33f92-ec9c-447a-ba57-7a2b31ffacc1"). InnerVolumeSpecName "kube-api-access-27654". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.777588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777552    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777590    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhb8\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-kube-api-access-wmhb8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777605    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777618    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777632    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777644    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4cfs\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-kube-api-access-s4cfs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777658    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777671    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777685    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777698    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777711    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777724    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777735    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27654\" (UniqueName: \"kubernetes.io/projected/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-kube-api-access-27654\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777748    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777761    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777772    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c909f7c3-a340-4e1c-b15a-ffe6132e9835-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777784    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777801    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777814    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c909f7c3-a340-4e1c-b15a-ffe6132e9835-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777827    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be33f92-ec9c-447a-ba57-7a2b31ffacc1-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.777897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.777841    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.856907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.856880    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:47:03.884436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.884401    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:47:03.888954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.888923    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-8w8jf"]
Nov 02 07:47:03.980153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980112    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980188    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980350    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980379    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980466    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980519    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.980633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980572    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980625    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980665    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.980780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980734    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980828    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\") pod \"09be35b7-df0b-45af-a266-609dbbdccdcc\" (UID: \"09be35b7-df0b-45af-a266-609dbbdccdcc\") "
Nov 02 07:47:03.980856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980845    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.981054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.980982    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.981177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981146    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:03.981518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981482    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.981518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981510    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.981717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981527    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.981717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981541    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.981717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.981555    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be35b7-df0b-45af-a266-609dbbdccdcc-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:03.983062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.982994    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:03.983473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.983447    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:03.983583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.983566    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:03.983748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:03.983724    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6" pod "09be35b7-df0b-45af-a266-609dbbdccdcc" (UID: "09be35b7-df0b-45af-a266-609dbbdccdcc"). InnerVolumeSpecName "kube-api-access-xmsf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:04.082717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.082636    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:04.082717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.082666    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmsf6\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-kube-api-access-xmsf6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:04.082717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.082676    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:04.082717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.082686    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be35b7-df0b-45af-a266-609dbbdccdcc-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:04.549475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.549441    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"
Nov 02 07:47:04.549475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.549464    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9" event={"ID":"5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c","Type":"ContainerDied","Data":"c8ad0cbee5bf1d32eb71d1f45796b070880d702fe931e5c62b6b202d04d12e8e"}
Nov 02 07:47:04.549984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.549525    4515 scope.go:122] "RemoveContainer" containerID="26a744d8cb119a4075a7a18a42010c1c8b951d0b60d8c2926665de6384533594"
Nov 02 07:47:04.552359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.552323    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"
Nov 02 07:47:04.552470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.552354    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh" event={"ID":"09be35b7-df0b-45af-a266-609dbbdccdcc","Type":"ContainerDied","Data":"31175583dd2ddb49a5493c7cc28a8862636113ebe03dc29a41c282ddb9755bf6"}
Nov 02 07:47:04.554061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.554035    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6" event={"ID":"c909f7c3-a340-4e1c-b15a-ffe6132e9835","Type":"ContainerDied","Data":"23c5712f7ddbfe4af39e8f1ad92fa74fcc3bbac53e248a3a3ec0db7d01941408"}
Nov 02 07:47:04.554061 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.554052    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"
Nov 02 07:47:04.563472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.563432    4515 scope.go:122] "RemoveContainer" containerID="cb4df52ea4c7c34728528c30aa4d5b95e556b5144beffb6405ce13fc37cb4951"
Nov 02 07:47:04.573244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.573225    4515 scope.go:122] "RemoveContainer" containerID="78c476639f03add6e6cb05614e3923261c0f9bcc608afd13d0126c5b9a3c17e2"
Nov 02 07:47:04.635962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.635927    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:47:04.648449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.648418    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-9h8xh"]
Nov 02 07:47:04.680521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.680491    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:47:04.686622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.686594    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-vcww6"]
Nov 02 07:47:04.726214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.726182    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:47:04.729897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:04.729873    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dhjw9"]
Nov 02 07:47:05.374711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:05.374675    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" path="/var/lib/kubelet/pods/09be35b7-df0b-45af-a266-609dbbdccdcc/volumes"
Nov 02 07:47:05.375193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:05.375180    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" path="/var/lib/kubelet/pods/5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c/volumes"
Nov 02 07:47:05.375641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:05.375628    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" path="/var/lib/kubelet/pods/8be33f92-ec9c-447a-ba57-7a2b31ffacc1/volumes"
Nov 02 07:47:05.376402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:05.376388    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" path="/var/lib/kubelet/pods/c909f7c3-a340-4e1c-b15a-ffe6132e9835/volumes"
Nov 02 07:47:08.483223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:08.483185    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:47:09.493779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:09.493751    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:47:10.530019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:10.529949    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:47:10.558331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:10.554562    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:47:13.836647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.836612    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:47:13.837178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837163    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy"
Nov 02 07:47:13.837224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837178    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy"
Nov 02 07:47:13.837224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837203    4515 cpu_manager.go:414] "containerMap: removing container" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy"
Nov 02 07:47:13.837224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837208    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy"
Nov 02 07:47:13.837224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837223    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837230    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837239    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837245    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837253    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837258    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837265    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" containerName="payload-pre-processing"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837269    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" containerName="payload-pre-processing"
Nov 02 07:47:13.837379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837380    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec1cfa4-0d0f-4acc-b993-291281fd96fd" containerName="payload-pre-processing"
Nov 02 07:47:13.837614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837393    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbbb7ea-71e6-4ffd-8feb-31efd80d4d5c" containerName="istio-proxy"
Nov 02 07:47:13.837614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837402    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="09be35b7-df0b-45af-a266-609dbbdccdcc" containerName="istio-proxy"
Nov 02 07:47:13.837614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837409    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.837614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837415    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c909f7c3-a340-4e1c-b15a-ffe6132e9835" containerName="istio-proxy"
Nov 02 07:47:13.837614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.837420    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be33f92-ec9c-447a-ba57-7a2b31ffacc1" containerName="istio-proxy"
Nov 02 07:47:13.841746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.841728    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.859922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.859894    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b7fde8-serving-cert\""
Nov 02 07:47:13.864442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.864411    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:47:13.888226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.888190    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.888462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.888417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsvf\" (UniqueName: \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.888578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.888551    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.888700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.888683    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.990070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.990039    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.990299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.990116    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.990299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.990180    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsvf\" (UniqueName: \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.990299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.990241    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.990840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.990819    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.992653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.992623    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:13.992773 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:13.992661    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:14.014104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:14.014076    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsvf\" (UniqueName: \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\") pod \"maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:14.159026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:14.158693    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:14.475741 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:14.475705    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a4b60b_187a_4a6b_8d3e_3de6eee097ff.slice/crio-40f5c51dac626ddda240ae42be7a16d1808a7adcd89ead81a3ce83e9796b8261 WatchSource:0}: Error finding container 40f5c51dac626ddda240ae42be7a16d1808a7adcd89ead81a3ce83e9796b8261: Status 404 returned error can't find the container with id 40f5c51dac626ddda240ae42be7a16d1808a7adcd89ead81a3ce83e9796b8261
Nov 02 07:47:14.476169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:14.476144    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:47:14.477321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:14.477294    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:47:14.750001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:14.749969    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" event={"ID":"86a4b60b-187a-4a6b-8d3e-3de6eee097ff","Type":"ContainerStarted","Data":"40f5c51dac626ddda240ae42be7a16d1808a7adcd89ead81a3ce83e9796b8261"}
Nov 02 07:47:15.758767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:15.758725    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" event={"ID":"86a4b60b-187a-4a6b-8d3e-3de6eee097ff","Type":"ContainerStarted","Data":"bef1e5d46264f8b04501574d699723b38680990d85f909f0b7657ed9c5fa4df2"}
Nov 02 07:47:15.759185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:15.758924    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:15.804436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:15.804386    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" podStartSLOduration=2.449054369 podStartE2EDuration="2.804372173s" totalImagesPullingTime="355.317804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:15.801400842 +0000 UTC m=+1380.563837749" watchObservedRunningTime="2026-11-02 07:47:15.804372173 +0000 UTC m=+1380.566809079"
Nov 02 07:47:17.561337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:17.551461    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:47:17.751959 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:17.751918    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:47:18.361304 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:18.361273    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:47:18.361589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:18.361561    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy" containerID="cri-o://20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5" gracePeriod=30
Nov 02 07:47:18.374107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:18.374078    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:47:18.374301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:18.374273    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy" containerID="cri-o://dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c" gracePeriod=30
Nov 02 07:47:19.070589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.070545    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:47:19.081775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.081738    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.089346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.089059    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:47:19.098581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.098550    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9f3716-openshift-default-dockercfg-4whw7\""
Nov 02 07:47:19.178493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178456    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178516    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178674    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178766 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178724    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44l9\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178784    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178823    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178845    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.178999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.178978    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.179035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.179017    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280234    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280287    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44l9\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280329    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280362    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280389    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280447    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280482    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280522    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280567    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280759    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280913    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.280989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.280972    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.281089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.281069    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.281162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.281088    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.282730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.282703    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.283045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.283026    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.303936 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.303901    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.304110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.304035    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44l9\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\") pod \"e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") " pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.410384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.406601    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:19.813395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.813370    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:47:19.815484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.815449    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:19.815602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.815530    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:19.815602 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.815579    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:19.821823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:19.821799    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerStarted","Data":"2f99daf47315026e09b27ab808244bc8522a0782de8b209b418988b1c76a2087"}
Nov 02 07:47:20.830411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:20.830371    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerStarted","Data":"88f39b68defefd59593caaa0021922e29e7207f28f1f71f5da8a5bba8c77113e"}
Nov 02 07:47:20.890345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:20.890275    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podStartSLOduration=1.890260043 podStartE2EDuration="1.890260043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:20.889042011 +0000 UTC m=+1385.651478917" watchObservedRunningTime="2026-11-02 07:47:20.890260043 +0000 UTC m=+1385.652697014"
Nov 02 07:47:21.408302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:21.408266    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:21.413766 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:21.413737    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:21.777733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:21.777707    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:47:21.839926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:21.839893    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:21.841025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:21.841006    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:47:23.667946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.667919    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:47:23.704958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.704928    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:47:23.851836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851802    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851851    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851876    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851906    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851937    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.851971    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852034    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852060    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852101    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852131    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852162    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852191    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852253    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852294    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852397    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852441    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852479    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852542    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852611    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\") pod \"405fbefd-42c7-48b0-b690-5449dbb6ec71\" (UID: \"405fbefd-42c7-48b0-b690-5449dbb6ec71\") "
Nov 02 07:47:23.852659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852654    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852671    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852703    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852744    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852780    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\") pod \"da05da06-700a-48ab-ae4a-c8849d2481ae\" (UID: \"da05da06-700a-48ab-ae4a-c8849d2481ae\") "
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852917    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.853170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.852461    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.853503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.853179    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.853840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.853810    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:23.853991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.853976    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405fbefd-42c7-48b0-b690-5449dbb6ec71-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854003    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854020    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854034    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05da06-700a-48ab-ae4a-c8849d2481ae-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854048    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854063    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854076    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854090    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854104    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.854569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.854117    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.855775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.855749    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.855775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.855767    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:23.855959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.855839    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.855959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.855867    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.856068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.855986    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "kube-api-access-phltv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.856068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.856019    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo" pod "405fbefd-42c7-48b0-b690-5449dbb6ec71" (UID: "405fbefd-42c7-48b0-b690-5449dbb6ec71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:23.856186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.856158    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "kube-api-access-tcddl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.857234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857212    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy" pod "da05da06-700a-48ab-ae4a-c8849d2481ae" (UID: "da05da06-700a-48ab-ae4a-c8849d2481ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.857601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857579    4515 generic.go:356] "Generic (PLEG): container finished" podID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerID="20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5" exitCode=0
Nov 02 07:47:23.857658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857634    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"
Nov 02 07:47:23.857691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857658    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerDied","Data":"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"}
Nov 02 07:47:23.857738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857712    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8" event={"ID":"da05da06-700a-48ab-ae4a-c8849d2481ae","Type":"ContainerDied","Data":"b744ce4a4ec9ded83ff32d2ed9fd64e712d74b4cfe87dfe45d9f861a7ce488ef"}
Nov 02 07:47:23.857775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.857743    4515 scope.go:122] "RemoveContainer" containerID="20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"
Nov 02 07:47:23.859182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.859156    4515 generic.go:356] "Generic (PLEG): container finished" podID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerID="dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c" exitCode=0
Nov 02 07:47:23.859261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.859191    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" event={"ID":"405fbefd-42c7-48b0-b690-5449dbb6ec71","Type":"ContainerDied","Data":"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"}
Nov 02 07:47:23.859261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.859240    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l" event={"ID":"405fbefd-42c7-48b0-b690-5449dbb6ec71","Type":"ContainerDied","Data":"d6a3f31e52831ac416fa8d7fb39c6c9c412fc3d46901ff466f8d47830637ff41"}
Nov 02 07:47:23.859261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.859196    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"
Nov 02 07:47:23.868129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.868032    4515 scope.go:122] "RemoveContainer" containerID="4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"
Nov 02 07:47:23.877316 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.877284    4515 scope.go:122] "RemoveContainer" containerID="20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"
Nov 02 07:47:23.877616 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:23.877594    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5\": container with ID starting with 20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5 not found: ID does not exist" containerID="20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"
Nov 02 07:47:23.877665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.877626    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5"} err="failed to get container status \"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5\": rpc error: code = NotFound desc = could not find container \"20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5\": container with ID starting with 20970ea472c37d5bbbfe495111502ccd88a783bb962f499d9f3e8e95afd0bce5 not found: ID does not exist"
Nov 02 07:47:23.877665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.877645    4515 scope.go:122] "RemoveContainer" containerID="4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"
Nov 02 07:47:23.877904 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:23.877877    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248\": container with ID starting with 4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248 not found: ID does not exist" containerID="4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"
Nov 02 07:47:23.877940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.877911    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248"} err="failed to get container status \"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248\": rpc error: code = NotFound desc = could not find container \"4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248\": container with ID starting with 4eda279b18b92dad2aee11232fecf30f9d3fdfc8672db2f6c542d7139260d248 not found: ID does not exist"
Nov 02 07:47:23.877940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.877925    4515 scope.go:122] "RemoveContainer" containerID="dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"
Nov 02 07:47:23.887674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.887645    4515 scope.go:122] "RemoveContainer" containerID="dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"
Nov 02 07:47:23.887994 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:23.887975    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c\": container with ID starting with dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c not found: ID does not exist" containerID="dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"
Nov 02 07:47:23.888064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.888007    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c"} err="failed to get container status \"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c\": rpc error: code = NotFound desc = could not find container \"dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c\": container with ID starting with dd799402b67532f129499117014bab46948e3755f4fcc78b4f10a970c80fa07c not found: ID does not exist"
Nov 02 07:47:23.912194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.912160    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:47:23.916727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.916696    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-xxc6l"]
Nov 02 07:47:23.955196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955154    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcddl\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-kube-api-access-tcddl\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955184    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955201    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955214    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phltv\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-kube-api-access-phltv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955228    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955240    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05da06-700a-48ab-ae4a-c8849d2481ae-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955252    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955264    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405fbefd-42c7-48b0-b690-5449dbb6ec71-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.955625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.955454    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:47:23.959420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:23.959395    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-mkbg8"]
Nov 02 07:47:25.144975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.144943    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:47:25.145534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145518    4515 cpu_manager.go:414] "containerMap: removing container" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.145585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145535    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.145585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145551    4515 cpu_manager.go:414] "containerMap: removing container" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.145585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145556    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.145585 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145585    4515 cpu_manager.go:414] "containerMap: removing container" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy"
Nov 02 07:47:25.145711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145591    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy"
Nov 02 07:47:25.145711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145670    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.145711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.145683    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" containerName="istio-proxy"
Nov 02 07:47:25.150675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.150654    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.168474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.168445    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:47:25.170143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.170116    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.170414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.170303    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdw9\" (UniqueName: \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.177721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.177695    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f3716-dockercfg-wr6j8\""
Nov 02 07:47:25.177721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.177700    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f3716\""
Nov 02 07:47:25.320333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.272192    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdw9\" (UniqueName: \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.320860 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.320827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.321253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.314168    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdw9\" (UniqueName: \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.321408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.305409    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:47:25.322581 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.322557    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" containerName="istio-proxy"
Nov 02 07:47:25.329396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.323170    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.350335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.348147    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:47:25.350335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.348294    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.391246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.391207    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405fbefd-42c7-48b0-b690-5449dbb6ec71" path="/var/lib/kubelet/pods/405fbefd-42c7-48b0-b690-5449dbb6ec71/volumes"
Nov 02 07:47:25.396902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.392563    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da05da06-700a-48ab-ae4a-c8849d2481ae" path="/var/lib/kubelet/pods/da05da06-700a-48ab-ae4a-c8849d2481ae/volumes"
Nov 02 07:47:25.423869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.422771    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkpw\" (UniqueName: \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.423869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.422876    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.463927 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.463893    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:25.525178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.525134    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkpw\" (UniqueName: \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.525285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.525223    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.526173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.526149    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.552351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.550822    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkpw\" (UniqueName: \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\") pod \"payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.709007 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.708915    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:25.800410 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:25.794863    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0631e31b_2f9d_47f4_86a0_49961d877efb.slice/crio-43ae3f34fac47a69c3915b752ed5c8c1f8b2d6528239f38282b17bf016edc5c5 WatchSource:0}: Error finding container 43ae3f34fac47a69c3915b752ed5c8c1f8b2d6528239f38282b17bf016edc5c5: Status 404 returned error can't find the container with id 43ae3f34fac47a69c3915b752ed5c8c1f8b2d6528239f38282b17bf016edc5c5
Nov 02 07:47:25.822732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.806903    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:47:25.878388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:25.878287    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" event={"ID":"0631e31b-2f9d-47f4-86a0-49961d877efb","Type":"ContainerStarted","Data":"43ae3f34fac47a69c3915b752ed5c8c1f8b2d6528239f38282b17bf016edc5c5"}
Nov 02 07:47:26.066721 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:26.066690    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf812e1_f9ac_42a4_aa50_00c44accb0f4.slice/crio-e8f3b85bdbc4b13ad2ef0b05f1a63813df59beaea8ba39faac6cca4b7a5b79a8 WatchSource:0}: Error finding container e8f3b85bdbc4b13ad2ef0b05f1a63813df59beaea8ba39faac6cca4b7a5b79a8: Status 404 returned error can't find the container with id e8f3b85bdbc4b13ad2ef0b05f1a63813df59beaea8ba39faac6cca4b7a5b79a8
Nov 02 07:47:26.067831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.067802    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:47:26.889961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.889920    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" event={"ID":"0631e31b-2f9d-47f4-86a0-49961d877efb","Type":"ContainerStarted","Data":"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"}
Nov 02 07:47:26.890444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.889993    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:26.891531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.891508    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" event={"ID":"abf812e1-f9ac-42a4-aa50-00c44accb0f4","Type":"ContainerStarted","Data":"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"}
Nov 02 07:47:26.891640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.891550    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" event={"ID":"abf812e1-f9ac-42a4-aa50-00c44accb0f4","Type":"ContainerStarted","Data":"e8f3b85bdbc4b13ad2ef0b05f1a63813df59beaea8ba39faac6cca4b7a5b79a8"}
Nov 02 07:47:26.891640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.891579    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:26.930340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.930279    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" podStartSLOduration=1.9302650190000001 podStartE2EDuration="1.930265019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:26.929914795 +0000 UTC m=+1391.692351701" watchObservedRunningTime="2026-11-02 07:47:26.930265019 +0000 UTC m=+1391.692701925"
Nov 02 07:47:26.971995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:26.971940    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" podStartSLOduration=1.971924715 podStartE2EDuration="1.971924715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:26.970003832 +0000 UTC m=+1391.732440740" watchObservedRunningTime="2026-11-02 07:47:26.971924715 +0000 UTC m=+1391.734361622"
Nov 02 07:47:29.240259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.240169    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:47:29.240717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.240436    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" podUID="d085b0b4-87a5-45c7-ac9e-c627df117592" containerName="payload-processing" containerID="cri-o://a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7" gracePeriod=30
Nov 02 07:47:29.273334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.273282    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:47:29.273586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.273559    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" podUID="a0daf4db-46d7-4436-bb10-36502f10797e" containerName="payload-pre-processing" containerID="cri-o://1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab" gracePeriod=30
Nov 02 07:47:29.788681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.788650    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:47:29.792005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.791985    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:47:29.913158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.913120    4515 generic.go:356] "Generic (PLEG): container finished" podID="a0daf4db-46d7-4436-bb10-36502f10797e" containerID="1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab" exitCode=0
Nov 02 07:47:29.913400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.913190    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"
Nov 02 07:47:29.913400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.913211    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" event={"ID":"a0daf4db-46d7-4436-bb10-36502f10797e","Type":"ContainerDied","Data":"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"}
Nov 02 07:47:29.913400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.913271    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6" event={"ID":"a0daf4db-46d7-4436-bb10-36502f10797e","Type":"ContainerDied","Data":"b2a8ba910e9b846d1b8a1e2c39247ef10d9b67751be01c2023bca7a81fac779c"}
Nov 02 07:47:29.913400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.913298    4515 scope.go:122] "RemoveContainer" containerID="1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"
Nov 02 07:47:29.915040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.915014    4515 generic.go:356] "Generic (PLEG): container finished" podID="d085b0b4-87a5-45c7-ac9e-c627df117592" containerID="a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7" exitCode=0
Nov 02 07:47:29.915191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.915050    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" event={"ID":"d085b0b4-87a5-45c7-ac9e-c627df117592","Type":"ContainerDied","Data":"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"}
Nov 02 07:47:29.915191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.915067    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"
Nov 02 07:47:29.915191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.915110    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7" event={"ID":"d085b0b4-87a5-45c7-ac9e-c627df117592","Type":"ContainerDied","Data":"1822f1b8a05f37d0622845cf489c9e6320bcb804930685ff6c2ed417a9317dde"}
Nov 02 07:47:29.924777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.924757    4515 scope.go:122] "RemoveContainer" containerID="1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"
Nov 02 07:47:29.925059 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:29.925039    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab\": container with ID starting with 1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab not found: ID does not exist" containerID="1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"
Nov 02 07:47:29.925114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.925068    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab"} err="failed to get container status \"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab\": rpc error: code = NotFound desc = could not find container \"1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab\": container with ID starting with 1e38e20014a392fecfc94a5081600adbd4c3101ae95b41a939c93e741d04d0ab not found: ID does not exist"
Nov 02 07:47:29.925114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.925086    4515 scope.go:122] "RemoveContainer" containerID="a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"
Nov 02 07:47:29.934318 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.934288    4515 scope.go:122] "RemoveContainer" containerID="a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"
Nov 02 07:47:29.934604 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:29.934586    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7\": container with ID starting with a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7 not found: ID does not exist" containerID="a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"
Nov 02 07:47:29.934647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.934612    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7"} err="failed to get container status \"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7\": rpc error: code = NotFound desc = could not find container \"a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7\": container with ID starting with a54c3d09a05da8e631183581fc7f885a65da15ede1fd4df564a8575a21f0b7a7 not found: ID does not exist"
Nov 02 07:47:29.959111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959087    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\" (UniqueName: \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\") pod \"d085b0b4-87a5-45c7-ac9e-c627df117592\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") "
Nov 02 07:47:29.959369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959350    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\") pod \"d085b0b4-87a5-45c7-ac9e-c627df117592\" (UID: \"d085b0b4-87a5-45c7-ac9e-c627df117592\") "
Nov 02 07:47:29.959453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959435    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\") pod \"a0daf4db-46d7-4436-bb10-36502f10797e\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") "
Nov 02 07:47:29.959509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959493    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\" (UniqueName: \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\") pod \"a0daf4db-46d7-4436-bb10-36502f10797e\" (UID: \"a0daf4db-46d7-4436-bb10-36502f10797e\") "
Nov 02 07:47:29.959728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959707    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume" pod "d085b0b4-87a5-45c7-ac9e-c627df117592" (UID: "d085b0b4-87a5-45c7-ac9e-c627df117592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:29.959828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.959805    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume" pod "a0daf4db-46d7-4436-bb10-36502f10797e" (UID: "a0daf4db-46d7-4436-bb10-36502f10797e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:29.960051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.960034    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d085b0b4-87a5-45c7-ac9e-c627df117592-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:29.960116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.960052    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0daf4db-46d7-4436-bb10-36502f10797e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:29.961429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.961406    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld" pod "d085b0b4-87a5-45c7-ac9e-c627df117592" (UID: "d085b0b4-87a5-45c7-ac9e-c627df117592"). InnerVolumeSpecName "kube-api-access-7v5ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:29.961709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:29.961691    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln" pod "a0daf4db-46d7-4436-bb10-36502f10797e" (UID: "a0daf4db-46d7-4436-bb10-36502f10797e"). InnerVolumeSpecName "kube-api-access-q62ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:30.070381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.065518    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62ln\" (UniqueName: \"kubernetes.io/projected/a0daf4db-46d7-4436-bb10-36502f10797e-kube-api-access-q62ln\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:30.070381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.065564    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5ld\" (UniqueName: \"kubernetes.io/projected/d085b0b4-87a5-45c7-ac9e-c627df117592-kube-api-access-7v5ld\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:30.262950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.262918    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:47:30.275213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.275189    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-6fbd798fd-l48h7"]
Nov 02 07:47:30.310614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.310585    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:47:30.315579 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:30.315553    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2cc1f-7ff4fc98c5-bvxj6"]
Nov 02 07:47:31.374875 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:31.374839    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0daf4db-46d7-4436-bb10-36502f10797e" path="/var/lib/kubelet/pods/a0daf4db-46d7-4436-bb10-36502f10797e/volumes"
Nov 02 07:47:31.375326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:31.375298    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d085b0b4-87a5-45c7-ac9e-c627df117592" path="/var/lib/kubelet/pods/d085b0b4-87a5-45c7-ac9e-c627df117592/volumes"
Nov 02 07:47:34.811608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.811567    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:47:34.812537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812511    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d085b0b4-87a5-45c7-ac9e-c627df117592" containerName="payload-processing"
Nov 02 07:47:34.812674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812538    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d085b0b4-87a5-45c7-ac9e-c627df117592" containerName="payload-processing"
Nov 02 07:47:34.812674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812586    4515 cpu_manager.go:414] "containerMap: removing container" podUID="a0daf4db-46d7-4436-bb10-36502f10797e" containerName="payload-pre-processing"
Nov 02 07:47:34.812674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812595    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0daf4db-46d7-4436-bb10-36502f10797e" containerName="payload-pre-processing"
Nov 02 07:47:34.812910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812732    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0daf4db-46d7-4436-bb10-36502f10797e" containerName="payload-pre-processing"
Nov 02 07:47:34.812910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.812757    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d085b0b4-87a5-45c7-ac9e-c627df117592" containerName="payload-processing"
Nov 02 07:47:34.818390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.818362    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:47:34.818522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.818467    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.847503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.847479    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d60b29-openshift-default-dockercfg-k6dph\""
Nov 02 07:47:34.852128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852100    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852157    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9cb\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852284    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852385    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852408    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852446 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852436    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852529    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852588    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.852662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.852612    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.959202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.959162    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.959524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.959504    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9cb\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.959702 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.959687    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.959827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.959814    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.959913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.959902    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.960021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.960009    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.960133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.960123    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.960246 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.960235    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.960366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.960351    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.961704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.961679    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.962110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.962090    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.962231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.962176    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.962822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.962793    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.987335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.963225    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.987571 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.967667    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:34.987696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:34.976160    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:35.042443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.035063    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9cb\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:35.052064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.051995    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\") pod \"e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") " pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:35.133410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.133328    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:35.508381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.506839    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:47:35.530291 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:35.530257    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289c2eb7_77ac_4258_9a07_bb383140ac56.slice/crio-aca20250ed4963dcc404a9dacc6727fb5fefc2a934dc5d3f1095d80fc3db1a0f WatchSource:0}: Error finding container aca20250ed4963dcc404a9dacc6727fb5fefc2a934dc5d3f1095d80fc3db1a0f: Status 404 returned error can't find the container with id aca20250ed4963dcc404a9dacc6727fb5fefc2a934dc5d3f1095d80fc3db1a0f
Nov 02 07:47:35.538814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.538766    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:35.538968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.538864    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:35.538968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:35.538909    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:36.066266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:36.066229    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerStarted","Data":"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"}
Nov 02 07:47:36.066670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:36.066287    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerStarted","Data":"aca20250ed4963dcc404a9dacc6727fb5fefc2a934dc5d3f1095d80fc3db1a0f"}
Nov 02 07:47:36.134164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:36.134132    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:36.139734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:36.139670    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podStartSLOduration=2.139653206 podStartE2EDuration="2.139653206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:36.138772084 +0000 UTC m=+1400.901208991" watchObservedRunningTime="2026-11-02 07:47:36.139653206 +0000 UTC m=+1400.902090117"
Nov 02 07:47:36.140678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:36.140654    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:37.071758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:37.071727    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:37.072890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:37.072873    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:47:37.899924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:37.899896    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:47:37.900114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:37.899973    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:47:39.966091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:39.966056    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:47:39.966664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:39.966284    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" containerName="maas-api" containerID="cri-o://cc46d5f46611f6095d8f412050bc7429c09c34552d02295bce72ff6ebf039824" gracePeriod=30
Nov 02 07:47:40.092964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.092938    4515 generic.go:356] "Generic (PLEG): container finished" podID="43ba0f21-acd5-474e-bc60-c58de68da073" containerID="cc46d5f46611f6095d8f412050bc7429c09c34552d02295bce72ff6ebf039824" exitCode=0
Nov 02 07:47:40.093067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.093011    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" event={"ID":"43ba0f21-acd5-474e-bc60-c58de68da073","Type":"ContainerDied","Data":"cc46d5f46611f6095d8f412050bc7429c09c34552d02295bce72ff6ebf039824"}
Nov 02 07:47:40.101371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.101337    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": dial tcp 10.133.0.108:8443: connect: connection refused"
Nov 02 07:47:40.136195 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.136161    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:47:40.136474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.136419    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" podUID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" containerName="payload-processing" containerID="cri-o://9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894" gracePeriod=30
Nov 02 07:47:40.267050 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.266993    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:47:40.367173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.367137    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\") pod \"43ba0f21-acd5-474e-bc60-c58de68da073\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") "
Nov 02 07:47:40.367393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.367188    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\") pod \"43ba0f21-acd5-474e-bc60-c58de68da073\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") "
Nov 02 07:47:40.367483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.367440    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\" (UniqueName: \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\") pod \"43ba0f21-acd5-474e-bc60-c58de68da073\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") "
Nov 02 07:47:40.367553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.367492    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca" pod "43ba0f21-acd5-474e-bc60-c58de68da073" (UID: "43ba0f21-acd5-474e-bc60-c58de68da073"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:40.367882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.367863    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\") pod \"43ba0f21-acd5-474e-bc60-c58de68da073\" (UID: \"43ba0f21-acd5-474e-bc60-c58de68da073\") "
Nov 02 07:47:40.368351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.368330    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba0f21-acd5-474e-bc60-c58de68da073-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.370438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.370411    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp" pod "43ba0f21-acd5-474e-bc60-c58de68da073" (UID: "43ba0f21-acd5-474e-bc60-c58de68da073"). InnerVolumeSpecName "kube-api-access-zxhgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:40.370539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.370472    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls" pod "43ba0f21-acd5-474e-bc60-c58de68da073" (UID: "43ba0f21-acd5-474e-bc60-c58de68da073"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:47:40.372431 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.372410    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs" pod "43ba0f21-acd5-474e-bc60-c58de68da073" (UID: "43ba0f21-acd5-474e-bc60-c58de68da073"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:47:40.405529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.405504    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:47:40.449979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.449946    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:47:40.450561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450545    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" containerName="maas-api"
Nov 02 07:47:40.450623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450562    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" containerName="maas-api"
Nov 02 07:47:40.450623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450587    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" containerName="payload-processing"
Nov 02 07:47:40.450623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450593    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" containerName="payload-processing"
Nov 02 07:47:40.450722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450677    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" containerName="payload-processing"
Nov 02 07:47:40.450722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.450690    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" containerName="maas-api"
Nov 02 07:47:40.455858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.455836    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.479215    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d60b29\""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.479397    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d60b29-dockercfg-hq787\""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.480245    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\") pod \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") "
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.480343    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\" (UniqueName: \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\") pod \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\" (UID: \"4fe10f7b-e0f6-45be-a25f-61b86be2adcf\") "
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.481058    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhgp\" (UniqueName: \"kubernetes.io/projected/43ba0f21-acd5-474e-bc60-c58de68da073-kube-api-access-zxhgp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.481086    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.481103    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ba0f21-acd5-474e-bc60-c58de68da073-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.485538    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf" pod "4fe10f7b-e0f6-45be-a25f-61b86be2adcf" (UID: "4fe10f7b-e0f6-45be-a25f-61b86be2adcf"). InnerVolumeSpecName "kube-api-access-whkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.485547    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume" pod "4fe10f7b-e0f6-45be-a25f-61b86be2adcf" (UID: "4fe10f7b-e0f6-45be-a25f-61b86be2adcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:40.497403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.487939    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:47:40.613414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.613374    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:47:40.677334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.621770    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.677334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.659774    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fc7\" (UniqueName: \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.677334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.659914    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.677334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.665359    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkgf\" (UniqueName: \"kubernetes.io/projected/4fe10f7b-e0f6-45be-a25f-61b86be2adcf-kube-api-access-whkgf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.766296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.766257    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.767558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.767423    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.767925 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.767907    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fc7\" (UniqueName: \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.795055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.794982    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fc7\" (UniqueName: \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\") pod \"payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:40.796983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.796458    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:47:40.796983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.796586    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.848412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.848373    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:47:40.856948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.856142    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.870822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.870785    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px57p\" (UniqueName: \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.870984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.870852    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwdp\" (UniqueName: \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.870984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.870887    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.870984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.870978    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.871157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.871074    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.871157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.871147    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.875586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.875539    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9f3716-serving-cert\""
Nov 02 07:47:40.911532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.909976    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:47:40.917721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.917689    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:47:40.922907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.922875    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.952974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.952942    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:47:40.973251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.973214    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.973814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.973788    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.973971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.973954    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.974121 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974104    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.974229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974217    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.974350 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974336    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.974595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974578    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.974742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974727    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px57p\" (UniqueName: \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.974857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974844    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwdp\" (UniqueName: \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.974953 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.974941    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.975063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.975049    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.975168 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.975155    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.975329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.975296    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.975501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.975484    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rzv\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:40.975705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.975688    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.976133 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:40.976111    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-serving-cert: secret "maas-api-e2e-ipp-a-9f3716-serving-cert" not found
Nov 02 07:47:40.976371 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.976345    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:40.976505 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:40.976378    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls podName:db23ab4f-340f-45b8-97d0-25333979675d nodeName:}" failed. No retries permitted until 2026-11-02 07:47:41.476354618 +0000 UTC m=+1406.238791506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls") pod "maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" (UID: "db23ab4f-340f-45b8-97d0-25333979675d") : secret "maas-api-e2e-ipp-a-9f3716-serving-cert" not found
Nov 02 07:47:40.976899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.976878    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b2023a-openshift-default-dockercfg-cxhj9\""
Nov 02 07:47:40.977614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.977590    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:40.981218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:40.981182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:41.002193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.002159    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwdp\" (UniqueName: \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\") pod \"payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:41.003838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.003813    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px57p\" (UniqueName: \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:41.077469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077373    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077480    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077521    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077536    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077584    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077634 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077622    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rzv\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077732    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077786    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.077908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077826    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.078065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077959    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.078065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.077999    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.078166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.078117    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.078248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.078230    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.078300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.078260    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.079935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.079904    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.080386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.080366    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.082303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.082278    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:41.101382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.101356    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.101923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.101836    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rzv\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\") pod \"e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") " pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.103114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.103084    4515 generic.go:356] "Generic (PLEG): container finished" podID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" containerID="9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894" exitCode=0
Nov 02 07:47:41.103227 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.103209    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" event={"ID":"4fe10f7b-e0f6-45be-a25f-61b86be2adcf","Type":"ContainerDied","Data":"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"}
Nov 02 07:47:41.103274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.103266    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq" event={"ID":"4fe10f7b-e0f6-45be-a25f-61b86be2adcf","Type":"ContainerDied","Data":"0b311b0964d5f616753872f03e8a2089ce3b02294993c8176d2704dd9e02bdb5"}
Nov 02 07:47:41.103410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.103296    4515 scope.go:122] "RemoveContainer" containerID="9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"
Nov 02 07:47:41.103533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.103509    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"
Nov 02 07:47:41.107016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.106984    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482" event={"ID":"43ba0f21-acd5-474e-bc60-c58de68da073","Type":"ContainerDied","Data":"f1d22e35e95f9b958fe649a987e2e54ed1272d606e0746ffe639e1d650362792"}
Nov 02 07:47:41.107137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.107118    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"
Nov 02 07:47:41.120566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.120538    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:41.166608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.165485    4515 scope.go:122] "RemoveContainer" containerID="9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"
Nov 02 07:47:41.171620 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:41.167013    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894\": container with ID starting with 9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894 not found: ID does not exist" containerID="9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"
Nov 02 07:47:41.171620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.167058    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894"} err="failed to get container status \"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894\": rpc error: code = NotFound desc = could not find container \"9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894\": container with ID starting with 9c655052ffa86f981f8f9f8c9fb5e605b8ecfc5506f90f630351c1ede336b894 not found: ID does not exist"
Nov 02 07:47:41.171620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.167086    4515 scope.go:122] "RemoveContainer" containerID="cc46d5f46611f6095d8f412050bc7429c09c34552d02295bce72ff6ebf039824"
Nov 02 07:47:41.268337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.260107    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:47:41.299141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.299110    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cc3f7-55f95bc477-zn482"]
Nov 02 07:47:41.303255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.300751    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:41.350622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.350551    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:47:41.376347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.374420    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-54d8776fcd-wsckq"]
Nov 02 07:47:41.391169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.391133    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ba0f21-acd5-474e-bc60-c58de68da073" path="/var/lib/kubelet/pods/43ba0f21-acd5-474e-bc60-c58de68da073/volumes"
Nov 02 07:47:41.392547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.392522    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe10f7b-e0f6-45be-a25f-61b86be2adcf" path="/var/lib/kubelet/pods/4fe10f7b-e0f6-45be-a25f-61b86be2adcf/volumes"
Nov 02 07:47:41.503872 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.503831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:41.507207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.507147    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:41.647429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.638231    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:47:41.676256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.676221    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:47:41.716971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.716944    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:47:41.723589 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:41.722723    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355898c2_3263_4bf9_a9a2_2c1f46997c28.slice/crio-4e879cc575ad76362a02b840b0aeee1e2e4e9b1f0e8a59065f7a7e47b1c973b6 WatchSource:0}: Error finding container 4e879cc575ad76362a02b840b0aeee1e2e4e9b1f0e8a59065f7a7e47b1c973b6: Status 404 returned error can't find the container with id 4e879cc575ad76362a02b840b0aeee1e2e4e9b1f0e8a59065f7a7e47b1c973b6
Nov 02 07:47:41.779901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.779874    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:41.912330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.912282    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:47:41.912587 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:41.912559    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264df5d7_7b99_4d40_9b6c_a3a4249a742c.slice/crio-a947db9895a091e8d2288a2788f2faeb29bb042a70fdf9888d88c4a647c9b341 WatchSource:0}: Error finding container a947db9895a091e8d2288a2788f2faeb29bb042a70fdf9888d88c4a647c9b341: Status 404 returned error can't find the container with id a947db9895a091e8d2288a2788f2faeb29bb042a70fdf9888d88c4a647c9b341
Nov 02 07:47:41.915415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.915380    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:41.915534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.915462    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:41.915534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:41.915511    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:42.117382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.117340    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" event={"ID":"355898c2-3263-4bf9-a9a2-2c1f46997c28","Type":"ContainerStarted","Data":"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"}
Nov 02 07:47:42.117847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.117411    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:42.117847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.117435    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" event={"ID":"355898c2-3263-4bf9-a9a2-2c1f46997c28","Type":"ContainerStarted","Data":"4e879cc575ad76362a02b840b0aeee1e2e4e9b1f0e8a59065f7a7e47b1c973b6"}
Nov 02 07:47:42.119655 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.119623    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" event={"ID":"3ba9cb59-5c5e-4e35-bccf-0339c8505179","Type":"ContainerStarted","Data":"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"}
Nov 02 07:47:42.119815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.119684    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:42.119815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.119711    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" event={"ID":"3ba9cb59-5c5e-4e35-bccf-0339c8505179","Type":"ContainerStarted","Data":"4865a76d32e105a060e0ec4e6f2afa7b5c3b944d9ef0233238ec7d6efcc5dc1a"}
Nov 02 07:47:42.121771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.121745    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerStarted","Data":"8007e6257ffbe6a24795c5b49de232c518c443099956da4185af4d66d2d7ad7c"}
Nov 02 07:47:42.121877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.121798    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerStarted","Data":"a947db9895a091e8d2288a2788f2faeb29bb042a70fdf9888d88c4a647c9b341"}
Nov 02 07:47:42.124382 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:42.124357    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb23ab4f_340f_45b8_97d0_25333979675d.slice/crio-1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6 WatchSource:0}: Error finding container 1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6: Status 404 returned error can't find the container with id 1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6
Nov 02 07:47:42.124519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.124491    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:47:42.161644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.161526    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" podStartSLOduration=2.161505479 podStartE2EDuration="2.161505479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:42.159980896 +0000 UTC m=+1406.922417804" watchObservedRunningTime="2026-11-02 07:47:42.161505479 +0000 UTC m=+1406.923942386"
Nov 02 07:47:42.204429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.204373    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" podStartSLOduration=2.204359686 podStartE2EDuration="2.204359686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:42.201799029 +0000 UTC m=+1406.964235935" watchObservedRunningTime="2026-11-02 07:47:42.204359686 +0000 UTC m=+1406.966796591"
Nov 02 07:47:42.252041 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.251974    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podStartSLOduration=2.251956492 podStartE2EDuration="2.251956492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:42.249247259 +0000 UTC m=+1407.011684166" watchObservedRunningTime="2026-11-02 07:47:42.251956492 +0000 UTC m=+1407.014393399"
Nov 02 07:47:42.301403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.301365    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:42.303153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.303127    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:47:42.303270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:42.303173    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:47:43.150106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:43.150065    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" event={"ID":"db23ab4f-340f-45b8-97d0-25333979675d","Type":"ContainerStarted","Data":"1013cafd25dc505611a6e57add65533491cb628f1245ef54f0d0d81f837a2aee"}
Nov 02 07:47:43.150600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:43.150130    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" event={"ID":"db23ab4f-340f-45b8-97d0-25333979675d","Type":"ContainerStarted","Data":"1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6"}
Nov 02 07:47:43.150600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:43.150158    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:43.207639 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:43.207582    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" podStartSLOduration=2.748383365 podStartE2EDuration="3.207566966s" totalImagesPullingTime="459.183601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:43.193064973 +0000 UTC m=+1407.955501880" watchObservedRunningTime="2026-11-02 07:47:43.207566966 +0000 UTC m=+1407.970003871"
Nov 02 07:47:43.306032 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:43.306002    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:44.163740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:44.161439    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:44.163740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:44.163718    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:47:46.591939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.591905    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:47:46.596920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.596885    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.616030 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.615989    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:47:46.633794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.633655    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b2023a-dockercfg-kj824\""
Nov 02 07:47:46.633794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.633760    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b2023a\""
Nov 02 07:47:46.844444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.808388    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:47:46.861408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.861246    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.861607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.861467    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzrd\" (UniqueName: \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.899334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.898647    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:46.911746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.911711    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:47:46.963995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.963949    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.964198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.964114    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:46.964198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.964162    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlbf\" (UniqueName: \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:46.964329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.964253    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzrd\" (UniqueName: \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.966464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.966432    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:46.988967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:46.988930    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzrd\" (UniqueName: \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\") pod \"payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:47.066143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.066105    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:47.066361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.066158    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlbf\" (UniqueName: \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:47.066837 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.066807    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:47.089861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.089833    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlbf\" (UniqueName: \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\") pod \"payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:47.216364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.216250    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:47.257105 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.257064    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:47.553004 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:47.552971    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e054c4_97a5_447a_a16f_15a004745833.slice/crio-5b45b5ba0aa70033f9749c52586c1af231a87aac82597002fcc7900e287648e8 WatchSource:0}: Error finding container 5b45b5ba0aa70033f9749c52586c1af231a87aac82597002fcc7900e287648e8: Status 404 returned error can't find the container with id 5b45b5ba0aa70033f9749c52586c1af231a87aac82597002fcc7900e287648e8
Nov 02 07:47:47.553992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.553966    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:47:47.609012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:47.608989    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:47:47.610128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:47.610102    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8fd132_3171_4e4e_b1ed_5ff515e83621.slice/crio-5b644e9ccfeb3e72d38531dd2c4a68c36a19710686b23030a4c2a66cf4a48b8b WatchSource:0}: Error finding container 5b644e9ccfeb3e72d38531dd2c4a68c36a19710686b23030a4c2a66cf4a48b8b: Status 404 returned error can't find the container with id 5b644e9ccfeb3e72d38531dd2c4a68c36a19710686b23030a4c2a66cf4a48b8b
Nov 02 07:47:47.847577 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:47:47.847533    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:47:48.191046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.190954    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" event={"ID":"80e054c4-97a5-447a-a16f-15a004745833","Type":"ContainerStarted","Data":"b5d2ea929df80cc6c5b9bc351992885683535c7ce2cd93fc7c6c97d620485ac1"}
Nov 02 07:47:48.191046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.191016    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" event={"ID":"80e054c4-97a5-447a-a16f-15a004745833","Type":"ContainerStarted","Data":"5b45b5ba0aa70033f9749c52586c1af231a87aac82597002fcc7900e287648e8"}
Nov 02 07:47:48.191284 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.191050    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:48.192617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.192590    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" event={"ID":"ba8fd132-3171-4e4e-b1ed-5ff515e83621","Type":"ContainerStarted","Data":"e3fd50398234caff08de646f26e7ce9c04b1e93c756d0cbc247e37a3d0f4a762"}
Nov 02 07:47:48.192750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.192648    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:48.192750 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.192673    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" event={"ID":"ba8fd132-3171-4e4e-b1ed-5ff515e83621","Type":"ContainerStarted","Data":"5b644e9ccfeb3e72d38531dd2c4a68c36a19710686b23030a4c2a66cf4a48b8b"}
Nov 02 07:47:48.233770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.233711    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" podStartSLOduration=2.233696601 podStartE2EDuration="2.233696601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:48.232260845 +0000 UTC m=+1412.994697752" watchObservedRunningTime="2026-11-02 07:47:48.233696601 +0000 UTC m=+1412.996133521"
Nov 02 07:47:48.281619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.281557    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" podStartSLOduration=2.281542841 podStartE2EDuration="2.281542841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:48.278605446 +0000 UTC m=+1413.041042355" watchObservedRunningTime="2026-11-02 07:47:48.281542841 +0000 UTC m=+1413.043979776"
Nov 02 07:47:48.844171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.844120    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:47:48.857640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.857607    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.875859 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.875827    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:47:48.895870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.895830    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-16e21e-openshift-default-dockercfg-wrfxv\""
Nov 02 07:47:48.922529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.922484    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.922731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.922550    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.922731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.922608    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.922731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.922722    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.922901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.922886    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvxz\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.923024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.923002    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.923107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.923092    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.923178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.923163    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:48.923219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:48.923207    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.009895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.009858    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:47:49.010143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.010117    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy" containerID="cri-o://116947f249777cfce8bf137e88b561364411af7e6f445da9835b4785be1b9b36" gracePeriod=30
Nov 02 07:47:49.024986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.024953    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025013    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025055    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025137    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvxz\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025230    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025328    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025400    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025441    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025449    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.025835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.025545    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.026518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.026493    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.026764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.026745    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.026974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.026937    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.027337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.027300    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.031499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.029985    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.031499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.030085    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.049920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.049888    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.050174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.050149    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvxz\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\") pod \"e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") " pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.216384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.190583    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:49.216384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.200495    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:47:49.624128 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:49.624088    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260dc4bc_11da_484b_9c1a_6f5d1f2c7a19.slice/crio-0927549de3273690ca1d2f752c0853b654ebaae973c26b5a04811c8ad6fca62f WatchSource:0}: Error finding container 0927549de3273690ca1d2f752c0853b654ebaae973c26b5a04811c8ad6fca62f: Status 404 returned error can't find the container with id 0927549de3273690ca1d2f752c0853b654ebaae973c26b5a04811c8ad6fca62f
Nov 02 07:47:49.625666 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.625631    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:47:49.626457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.626425    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:49.626567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.626489    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:49.626567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:49.626516    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:50.223465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:50.223423    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerStarted","Data":"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"}
Nov 02 07:47:50.223882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:50.223512    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerStarted","Data":"0927549de3273690ca1d2f752c0853b654ebaae973c26b5a04811c8ad6fca62f"}
Nov 02 07:47:50.287873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:50.287794    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podStartSLOduration=2.287770994 podStartE2EDuration="2.287770994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:50.270541633 +0000 UTC m=+1415.032978539" watchObservedRunningTime="2026-11-02 07:47:50.287770994 +0000 UTC m=+1415.050207901"
Nov 02 07:47:51.191775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:51.191733    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:51.197915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:51.197886    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:51.230537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:51.230505    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:51.231595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:51.231576    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:47:52.402784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:52.401929    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": read tcp 10.133.0.2:41296->10.133.0.124:15021: read: connection reset by peer" start-of-body=
Nov 02 07:47:52.402784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:52.402003    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": read tcp 10.133.0.2:41296->10.133.0.124:15021: read: connection reset by peer"
Nov 02 07:47:53.153368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.153329    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:47:53.153548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.153398    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:47:53.247450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.247417    4515 generic.go:356] "Generic (PLEG): container finished" podID="e1468920-9ced-4001-a0e4-4721f031c853" containerID="12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096" exitCode=0
Nov 02 07:47:53.247636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.247487    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerDied","Data":"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"}
Nov 02 07:47:53.247951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.247933    4515 scope.go:122] "RemoveContainer" containerID="12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"
Nov 02 07:47:53.252764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.249588    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:53.252764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.249676    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:53.252764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:53.249719    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:47:54.261238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.261205    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerStarted","Data":"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"}
Nov 02 07:47:54.263649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.263625    4515 generic.go:356] "Generic (PLEG): container finished" podID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerID="116947f249777cfce8bf137e88b561364411af7e6f445da9835b4785be1b9b36" exitCode=0
Nov 02 07:47:54.263779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.263665    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerDied","Data":"116947f249777cfce8bf137e88b561364411af7e6f445da9835b4785be1b9b36"}
Nov 02 07:47:54.263779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.263717    4515 scope.go:122] "RemoveContainer" containerID="306be1d1973053077e37e72c2565a45d0bbef94211835b69d08c0b8fa49e77dc"
Nov 02 07:47:54.348197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.348169    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:47:54.451822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.451788    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.451865    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.451895    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452029 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.451941    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452059    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452124    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:54.452215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452267    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:54.452406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452286    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452367    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452370    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:54.452688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452425    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\") pod \"43345e75-8a66-4e3e-8fb1-bfce68636bed\" (UID: \"43345e75-8a66-4e3e-8fb1-bfce68636bed\") "
Nov 02 07:47:54.452688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452554    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:54.452802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.452776    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:54.453394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.453369    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.453559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.453408    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.453559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.453423    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.453559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.453436    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43345e75-8a66-4e3e-8fb1-bfce68636bed-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.453559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.453448    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.454830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.454803    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:54.454930 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.454890    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "kube-api-access-k476v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:54.455026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.455008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:54.455092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.455060    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy" pod "43345e75-8a66-4e3e-8fb1-bfce68636bed" (UID: "43345e75-8a66-4e3e-8fb1-bfce68636bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:54.555228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.555137    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k476v\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-kube-api-access-k476v\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.555228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.555170    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.555228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.555183    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.555228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.555195    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43345e75-8a66-4e3e-8fb1-bfce68636bed-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:54.645762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.645719    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:47:54.646788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.646760    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.646950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.646790    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.646999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.646973    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.646999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.646991    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.651019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.650996    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.654695 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.654672    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:47:54.666434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.666390    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-16e21e-dockercfg-cqnkw\""
Nov 02 07:47:54.666434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.666388    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-16e21e\""
Nov 02 07:47:54.802333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.790570    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:47:54.802333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.792807    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grq8r\" (UniqueName: \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.802333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.793003    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.819380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.809301    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.819380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.809349    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" containerName="istio-proxy"
Nov 02 07:47:54.859341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.856765    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:54.895674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.895632    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:47:54.914334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.914268    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.914541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.914487    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54nr\" (UniqueName: \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:54.926331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.914611    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:54.926331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.914748    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grq8r\" (UniqueName: \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.972748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.972709    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:54.978397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:54.978366    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grq8r\" (UniqueName: \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:55.022331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.016651    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54nr\" (UniqueName: \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:55.022331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.016768    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:55.022331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.017688    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:55.045877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.045757    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54nr\" (UniqueName: \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\") pod \"payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:55.188229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.188135    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:47:55.188427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.188383    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 07:47:55.188507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.188436    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 07:47:55.270384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.270353    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:55.271133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.271112    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"
Nov 02 07:47:55.271526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.271504    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6" event={"ID":"43345e75-8a66-4e3e-8fb1-bfce68636bed","Type":"ContainerDied","Data":"f3d883fff740832dc10732f75b1751f9a8b43417701050b614f4ee57d373b86d"}
Nov 02 07:47:55.271599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.271550    4515 scope.go:122] "RemoveContainer" containerID="116947f249777cfce8bf137e88b561364411af7e6f445da9835b4785be1b9b36"
Nov 02 07:47:55.288999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.288970    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:55.353240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.352403    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:47:55.356340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.356294    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cc3f7-openshift-default-5cbd84bc5-5s4f6"]
Nov 02 07:47:55.393536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.392052    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43345e75-8a66-4e3e-8fb1-bfce68636bed" path="/var/lib/kubelet/pods/43345e75-8a66-4e3e-8fb1-bfce68636bed/volumes"
Nov 02 07:47:55.608945 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:55.608904    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9664ac_03b6_4e78_956d_689790d5422a.slice/crio-d3785f623fb22fa7eb4730b85548f799d6c08993c925b5ba8b6e08f246399a97 WatchSource:0}: Error finding container d3785f623fb22fa7eb4730b85548f799d6c08993c925b5ba8b6e08f246399a97: Status 404 returned error can't find the container with id d3785f623fb22fa7eb4730b85548f799d6c08993c925b5ba8b6e08f246399a97
Nov 02 07:47:55.610608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.610584    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:47:55.656877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.656684    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:47:55.811893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.811855    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:47:55.812124 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.812100    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" podUID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" containerName="payload-processing" containerID="cri-o://993f3f7b6da282b2ccca6c3b72f669bf1acd0a7ef6a8f0196fc3dad1c3297ea6" gracePeriod=30
Nov 02 07:47:55.855864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.845006    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:47:55.855864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:55.845415    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" podUID="1925c757-6df8-4f9f-95c2-650b36fb3296" containerName="payload-pre-processing" containerID="cri-o://d514fb5980c82d47b99910cab0b5e205d8c1c74c01644c397465bd74bd885bb8" gracePeriod=30
Nov 02 07:47:56.222332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.222205    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 07:47:56.222332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.222267    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 07:47:56.463355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.463095    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" event={"ID":"4c9664ac-03b6-4e78-956d-689790d5422a","Type":"ContainerStarted","Data":"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"}
Nov 02 07:47:56.463355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.463147    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" event={"ID":"4c9664ac-03b6-4e78-956d-689790d5422a","Type":"ContainerStarted","Data":"d3785f623fb22fa7eb4730b85548f799d6c08993c925b5ba8b6e08f246399a97"}
Nov 02 07:47:56.463835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.463416    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:47:56.466050 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.466021    4515 generic.go:356] "Generic (PLEG): container finished" podID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" containerID="993f3f7b6da282b2ccca6c3b72f669bf1acd0a7ef6a8f0196fc3dad1c3297ea6" exitCode=0
Nov 02 07:47:56.466184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.466081    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" event={"ID":"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea","Type":"ContainerDied","Data":"993f3f7b6da282b2ccca6c3b72f669bf1acd0a7ef6a8f0196fc3dad1c3297ea6"}
Nov 02 07:47:56.468762 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.468736    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" event={"ID":"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48","Type":"ContainerStarted","Data":"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"}
Nov 02 07:47:56.468868 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.468778    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" event={"ID":"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48","Type":"ContainerStarted","Data":"d23fa1de251be031f91883c243e92e7a7dd410a8cc6f39bd6d3b9e5f6d456612"}
Nov 02 07:47:56.469200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.469178    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:47:56.471840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.471823    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:47:56.473544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.473507    4515 generic.go:356] "Generic (PLEG): container finished" podID="1925c757-6df8-4f9f-95c2-650b36fb3296" containerID="d514fb5980c82d47b99910cab0b5e205d8c1c74c01644c397465bd74bd885bb8" exitCode=0
Nov 02 07:47:56.473665 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.473571    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" event={"ID":"1925c757-6df8-4f9f-95c2-650b36fb3296","Type":"ContainerDied","Data":"d514fb5980c82d47b99910cab0b5e205d8c1c74c01644c397465bd74bd885bb8"}
Nov 02 07:47:56.477499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.477471    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:47:56.506562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.506512    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" podStartSLOduration=2.50649745 podStartE2EDuration="2.50649745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:56.503843935 +0000 UTC m=+1421.266280840" watchObservedRunningTime="2026-11-02 07:47:56.50649745 +0000 UTC m=+1421.268934380"
Nov 02 07:47:56.581094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.581056    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\") pod \"1925c757-6df8-4f9f-95c2-650b36fb3296\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") "
Nov 02 07:47:56.581302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.581262    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\" (UniqueName: \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\") pod \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") "
Nov 02 07:47:56.581441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.581354    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\" (UniqueName: \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\") pod \"1925c757-6df8-4f9f-95c2-650b36fb3296\" (UID: \"1925c757-6df8-4f9f-95c2-650b36fb3296\") "
Nov 02 07:47:56.581441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.581399    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\") pod \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\" (UID: \"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea\") "
Nov 02 07:47:56.582893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.582773    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume" pod "1925c757-6df8-4f9f-95c2-650b36fb3296" (UID: "1925c757-6df8-4f9f-95c2-650b36fb3296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:56.583567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.583540    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume" pod "440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" (UID: "440a7324-e2c4-4c54-8bbf-0a4e9b1398ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:56.588114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.585913    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9" pod "440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" (UID: "440a7324-e2c4-4c54-8bbf-0a4e9b1398ea"). InnerVolumeSpecName "kube-api-access-hw7l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:56.591043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.591019    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp" pod "1925c757-6df8-4f9f-95c2-650b36fb3296" (UID: "1925c757-6df8-4f9f-95c2-650b36fb3296"). InnerVolumeSpecName "kube-api-access-847hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:56.655343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.641879    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" podStartSLOduration=2.641857783 podStartE2EDuration="2.641857783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:56.64067159 +0000 UTC m=+1421.403108571" watchObservedRunningTime="2026-11-02 07:47:56.641857783 +0000 UTC m=+1421.404294690"
Nov 02 07:47:56.867410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.707076    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1925c757-6df8-4f9f-95c2-650b36fb3296-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:56.867410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.800411    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7l9\" (UniqueName: \"kubernetes.io/projected/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-kube-api-access-hw7l9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:56.867410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.800434    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847hp\" (UniqueName: \"kubernetes.io/projected/1925c757-6df8-4f9f-95c2-650b36fb3296-kube-api-access-847hp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:56.867410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.800450    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:47:56.894121 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.894085    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:47:56.895363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.895336    4515 cpu_manager.go:414] "containerMap: removing container" podUID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" containerName="payload-processing"
Nov 02 07:47:56.895528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.895510    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" containerName="payload-processing"
Nov 02 07:47:56.895685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.895672    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1925c757-6df8-4f9f-95c2-650b36fb3296" containerName="payload-pre-processing"
Nov 02 07:47:56.895769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.895758    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1925c757-6df8-4f9f-95c2-650b36fb3296" containerName="payload-pre-processing"
Nov 02 07:47:56.895976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.895963    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1925c757-6df8-4f9f-95c2-650b36fb3296" containerName="payload-pre-processing"
Nov 02 07:47:56.896097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.896086    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" containerName="payload-processing"
Nov 02 07:47:56.980336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.979398    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:47:56.980336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.979534    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:56.998430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:56.998404    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d60b29-serving-cert\""
Nov 02 07:47:57.116287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.113110    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnw4\" (UniqueName: \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.116287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.113462    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.116287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.113508    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.116287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.113551    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.188420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.188341    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 07:47:57.188420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.188396    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 07:47:57.215167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.215134    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.215392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.215176    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.215392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.215202    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.215392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.215244    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnw4\" (UniqueName: \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.215914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.215888    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.243336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.241926    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.243336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.243094    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.247200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.247173    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnw4\" (UniqueName: \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\") pod \"maas-api-e2e-praxis-a-d60b29-db67969db-p47hb\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.302882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.302845    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:57.481990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.481904    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"
Nov 02 07:47:57.482659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.482627    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5" event={"ID":"440a7324-e2c4-4c54-8bbf-0a4e9b1398ea","Type":"ContainerDied","Data":"d34fdfefdb0ea8f1d0fc1cc274b7be4e74af42c0a7b31e12207120ba2890d853"}
Nov 02 07:47:57.482778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.482704    4515 scope.go:122] "RemoveContainer" containerID="993f3f7b6da282b2ccca6c3b72f669bf1acd0a7ef6a8f0196fc3dad1c3297ea6"
Nov 02 07:47:57.485388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.485359    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77" event={"ID":"1925c757-6df8-4f9f-95c2-650b36fb3296","Type":"ContainerDied","Data":"864113feb34965efc46ffb721f1a1547f49071c7b8abba79138db21c0aef1de4"}
Nov 02 07:47:57.485499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.485416    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"
Nov 02 07:47:57.494360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.494339    4515 scope.go:122] "RemoveContainer" containerID="d514fb5980c82d47b99910cab0b5e205d8c1c74c01644c397465bd74bd885bb8"
Nov 02 07:47:57.544532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.544503    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:47:57.548012 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.547986    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-56bdf69b9b-vxfq5"]
Nov 02 07:47:57.581156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.581122    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:47:57.584650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.584622    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-85c7dcb6c-5jz77"]
Nov 02 07:47:57.645467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:57.645353    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:47:57.646662 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:47:57.646631    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b5c0f2_130d_4acd_a1ee_7f17438d2a5e.slice/crio-88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10 WatchSource:0}: Error finding container 88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10: Status 404 returned error can't find the container with id 88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10
Nov 02 07:47:58.208361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.201594    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 07:47:58.208361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.201731    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 07:47:58.515185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.495719    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" event={"ID":"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e","Type":"ContainerStarted","Data":"c3e10f077ceb3ca1ea5880f4cb551ada92009eb68e02a5651ff978e8c88d0254"}
Nov 02 07:47:58.515185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.495812    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" event={"ID":"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e","Type":"ContainerStarted","Data":"88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10"}
Nov 02 07:47:58.515185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.497327    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:47:58.548258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:58.547480    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" podStartSLOduration=2.144610135 podStartE2EDuration="2.547465558s" totalImagesPullingTime="402.855423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:47:58.546677766 +0000 UTC m=+1423.309114672" watchObservedRunningTime="2026-11-02 07:47:58.547465558 +0000 UTC m=+1423.309902489"
Nov 02 07:47:59.189190 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.189153    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 07:47:59.189404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.189208    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 07:47:59.244433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.244401    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:47:59.244636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.244564    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:47:59.374460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.374426    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1925c757-6df8-4f9f-95c2-650b36fb3296" path="/var/lib/kubelet/pods/1925c757-6df8-4f9f-95c2-650b36fb3296/volumes"
Nov 02 07:47:59.375092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:47:59.375073    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440a7324-e2c4-4c54-8bbf-0a4e9b1398ea" path="/var/lib/kubelet/pods/440a7324-e2c4-4c54-8bbf-0a4e9b1398ea/volumes"
Nov 02 07:48:00.192462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:00.192429    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:48:00.192838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:00.192715    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:48:00.193591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:00.193574    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:48:02.385291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.385260    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:48:02.391017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.390996    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.432712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.425614    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b2023a-serving-cert\""
Nov 02 07:48:02.432712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.431016    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:48:02.527927 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.527889    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.528112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.527945    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.528112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.527974    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhv6j\" (UniqueName: \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.528250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.528123    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.644370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.644259    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.645520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.645495    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.645745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.645723    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhv6j\" (UniqueName: \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.645940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.645923    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.660867 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:02.660833    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-serving-cert: secret "maas-api-e2e-praxis-b-b2023a-serving-cert" not found
Nov 02 07:48:02.661175 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:02.661157    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls podName:826818f4-110a-451e-81bf-bccb21729021 nodeName:}" failed. No retries permitted until 2026-11-02 07:48:03.16112304 +0000 UTC m=+1427.923559942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls") pod "maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" (UID: "826818f4-110a-451e-81bf-bccb21729021") : secret "maas-api-e2e-praxis-b-b2023a-serving-cert" not found
Nov 02 07:48:02.661543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.645398    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.664129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.664101    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:02.693867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:02.693820    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhv6j\" (UniqueName: \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:03.253800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.253762    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:03.256289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.256262    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:03.306433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.306394    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:03.616681 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:03.616037    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826818f4_110a_451e_81bf_bccb21729021.slice/crio-0e6079906217313ba8349e712fb2202b1b46f1bc37d529e546c34b9d3ad60239 WatchSource:0}: Error finding container 0e6079906217313ba8349e712fb2202b1b46f1bc37d529e546c34b9d3ad60239: Status 404 returned error can't find the container with id 0e6079906217313ba8349e712fb2202b1b46f1bc37d529e546c34b9d3ad60239
Nov 02 07:48:03.618788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.618762    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:48:03.875756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.875662    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:48:03.892040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:03.892010    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:48:04.586370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:04.585357    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" event={"ID":"826818f4-110a-451e-81bf-bccb21729021","Type":"ContainerStarted","Data":"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"}
Nov 02 07:48:04.586370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:04.585422    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" event={"ID":"826818f4-110a-451e-81bf-bccb21729021","Type":"ContainerStarted","Data":"0e6079906217313ba8349e712fb2202b1b46f1bc37d529e546c34b9d3ad60239"}
Nov 02 07:48:04.587352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:04.586720    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:05.509333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:05.508769    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" podStartSLOduration=3.211962123 podStartE2EDuration="3.508746882s" totalImagesPullingTime="296.784759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:04.66227645 +0000 UTC m=+1429.424713356" watchObservedRunningTime="2026-11-02 07:48:05.508746882 +0000 UTC m=+1430.271183791"
Nov 02 07:48:05.515333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:05.512813    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:48:05.551560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:05.551531    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:48:07.052382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.049330    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:48:07.061047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.060961    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.062857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.062826    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:48:07.076318 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.076272    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-894bbb-openshift-default-dockercfg-z6q95\""
Nov 02 07:48:07.160965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.160926    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.160984    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161084    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161161    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161232    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161268    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161292    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54922\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161399    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.161496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.161455    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.275441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.272933    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.275441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.273002    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.275441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.273124    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.275441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.273180    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.275441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.273239    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.284331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.281575    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.289512    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.289600    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.289650    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.289685    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54922\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.290434    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.290742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.290712    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.291711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.291683    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.292183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.292122    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.300680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.300597    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.326031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.325953    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.352330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.351131    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54922\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.355914 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.355839    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\") pod \"e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") " pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.403676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.403638    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:07.505055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.505000    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:48:07.507626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.507596    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:48:07.857532 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:07.857452    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad88fc9a_9940_420d_8e60_4fe5c699ff42.slice/crio-4387e76c2a26f29e92d075bdf9e3b36d8c820a5885bf04acd63113a4f699edaa WatchSource:0}: Error finding container 4387e76c2a26f29e92d075bdf9e3b36d8c820a5885bf04acd63113a4f699edaa: Status 404 returned error can't find the container with id 4387e76c2a26f29e92d075bdf9e3b36d8c820a5885bf04acd63113a4f699edaa
Nov 02 07:48:07.858224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.858201    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:48:07.860189 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.860156    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:07.860270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.860225    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:07.860270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:07.860256    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:08.665880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:08.665843    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerStarted","Data":"db12f1c23ae5f698e6142b7d53b8b32bf9f9b786308144787fc7faddbdc76142"}
Nov 02 07:48:08.666402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:08.665909    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerStarted","Data":"4387e76c2a26f29e92d075bdf9e3b36d8c820a5885bf04acd63113a4f699edaa"}
Nov 02 07:48:08.736334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:08.735614    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podStartSLOduration=2.735594732 podStartE2EDuration="2.735594732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:08.734824077 +0000 UTC m=+1433.497260984" watchObservedRunningTime="2026-11-02 07:48:08.735594732 +0000 UTC m=+1433.498031638"
Nov 02 07:48:08.879610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:08.879385    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" podUID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" containerName="payload-processing" containerID="cri-o://d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d" gracePeriod=25
Nov 02 07:48:08.903990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:08.903932    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" podUID="0631e31b-2f9d-47f4-86a0-49961d877efb" containerName="payload-pre-processing" containerID="cri-o://fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b" gracePeriod=25
Nov 02 07:48:09.362533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.362502    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:48:09.393155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.393118    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:48:09.404916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.404890    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:09.411858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.411833    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.486464    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\" (UniqueName: \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\") pod \"0631e31b-2f9d-47f4-86a0-49961d877efb\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") "
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.486660    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\") pod \"0631e31b-2f9d-47f4-86a0-49961d877efb\" (UID: \"0631e31b-2f9d-47f4-86a0-49961d877efb\") "
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.486753    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\" (UniqueName: \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\") pod \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") "
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.486827    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\") pod \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\" (UID: \"abf812e1-f9ac-42a4-aa50-00c44accb0f4\") "
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.488746    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume" pod "0631e31b-2f9d-47f4-86a0-49961d877efb" (UID: "0631e31b-2f9d-47f4-86a0-49961d877efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:09.496329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.489225    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume" pod "abf812e1-f9ac-42a4-aa50-00c44accb0f4" (UID: "abf812e1-f9ac-42a4-aa50-00c44accb0f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:09.499915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.498592    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw" pod "abf812e1-f9ac-42a4-aa50-00c44accb0f4" (UID: "abf812e1-f9ac-42a4-aa50-00c44accb0f4"). InnerVolumeSpecName "kube-api-access-8gkpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:09.500816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.500787    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9" pod "0631e31b-2f9d-47f4-86a0-49961d877efb" (UID: "0631e31b-2f9d-47f4-86a0-49961d877efb"). InnerVolumeSpecName "kube-api-access-zxdw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:09.588938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.588855    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkpw\" (UniqueName: \"kubernetes.io/projected/abf812e1-f9ac-42a4-aa50-00c44accb0f4-kube-api-access-8gkpw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.588938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.588889    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf812e1-f9ac-42a4-aa50-00c44accb0f4-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.588938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.588899    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdw9\" (UniqueName: \"kubernetes.io/projected/0631e31b-2f9d-47f4-86a0-49961d877efb-kube-api-access-zxdw9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.588938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.588908    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0631e31b-2f9d-47f4-86a0-49961d877efb-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.673335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.673271    4515 generic.go:356] "Generic (PLEG): container finished" podID="0631e31b-2f9d-47f4-86a0-49961d877efb" containerID="fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b" exitCode=0
Nov 02 07:48:09.673764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.673349    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" event={"ID":"0631e31b-2f9d-47f4-86a0-49961d877efb","Type":"ContainerDied","Data":"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"}
Nov 02 07:48:09.673764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.673393    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"
Nov 02 07:48:09.673764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.673404    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v" event={"ID":"0631e31b-2f9d-47f4-86a0-49961d877efb","Type":"ContainerDied","Data":"43ae3f34fac47a69c3915b752ed5c8c1f8b2d6528239f38282b17bf016edc5c5"}
Nov 02 07:48:09.673764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.673433    4515 scope.go:122] "RemoveContainer" containerID="fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"
Nov 02 07:48:09.674969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.674946    4515 generic.go:356] "Generic (PLEG): container finished" podID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" containerID="d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d" exitCode=0
Nov 02 07:48:09.675076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.675004    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"
Nov 02 07:48:09.675076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.675051    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" event={"ID":"abf812e1-f9ac-42a4-aa50-00c44accb0f4","Type":"ContainerDied","Data":"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"}
Nov 02 07:48:09.675201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.675104    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb" event={"ID":"abf812e1-f9ac-42a4-aa50-00c44accb0f4","Type":"ContainerDied","Data":"e8f3b85bdbc4b13ad2ef0b05f1a63813df59beaea8ba39faac6cca4b7a5b79a8"}
Nov 02 07:48:09.675565 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.675547    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:09.676802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.676778    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:48:09.684193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.684175    4515 scope.go:122] "RemoveContainer" containerID="fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"
Nov 02 07:48:09.684545 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:09.684524    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b\": container with ID starting with fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b not found: ID does not exist" containerID="fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"
Nov 02 07:48:09.684641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.684555    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b"} err="failed to get container status \"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b\": rpc error: code = NotFound desc = could not find container \"fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b\": container with ID starting with fb22b84f43192dca7367f5c395a908435c9ed8a2b650dcd03d68162225c1b46b not found: ID does not exist"
Nov 02 07:48:09.684641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.684574    4515 scope.go:122] "RemoveContainer" containerID="d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"
Nov 02 07:48:09.693931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.693913    4515 scope.go:122] "RemoveContainer" containerID="d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"
Nov 02 07:48:09.694197 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:09.694177    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d\": container with ID starting with d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d not found: ID does not exist" containerID="d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"
Nov 02 07:48:09.694260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.694203    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d"} err="failed to get container status \"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d\": rpc error: code = NotFound desc = could not find container \"d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d\": container with ID starting with d3d9341c28182ec11e9ab5dece794d4ce672a8b9dee73ab191876b52d3420f1d not found: ID does not exist"
Nov 02 07:48:09.779586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.779519    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:48:09.783698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.783666    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-7fcdbcfb99-tdn8v"]
Nov 02 07:48:09.815372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.815337    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:48:09.844717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:09.844537    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-78bc6c89df-jvdjb"]
Nov 02 07:48:11.415334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:11.414271    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0631e31b-2f9d-47f4-86a0-49961d877efb" path="/var/lib/kubelet/pods/0631e31b-2f9d-47f4-86a0-49961d877efb/volumes"
Nov 02 07:48:11.428337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:11.427111    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" path="/var/lib/kubelet/pods/abf812e1-f9ac-42a4-aa50-00c44accb0f4/volumes"
Nov 02 07:48:11.646178 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:11.645865    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:48:13.032013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.031978    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:48:13.032656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032638    4515 cpu_manager.go:414] "containerMap: removing container" podUID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" containerName="payload-processing"
Nov 02 07:48:13.032656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032656    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" containerName="payload-processing"
Nov 02 07:48:13.032791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032677    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0631e31b-2f9d-47f4-86a0-49961d877efb" containerName="payload-pre-processing"
Nov 02 07:48:13.032791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032682    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0631e31b-2f9d-47f4-86a0-49961d877efb" containerName="payload-pre-processing"
Nov 02 07:48:13.032791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032785    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf812e1-f9ac-42a4-aa50-00c44accb0f4" containerName="payload-processing"
Nov 02 07:48:13.032954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.032794    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0631e31b-2f9d-47f4-86a0-49961d877efb" containerName="payload-pre-processing"
Nov 02 07:48:13.037782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.037757    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:48:13.037918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.037872    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.041898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.041877    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.056219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.056198    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-16e21e-serving-cert\""
Nov 02 07:48:13.061342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.060487    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-894bbb\""
Nov 02 07:48:13.063014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.062988    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:48:13.097751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.097719    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:48:13.144087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.144050    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:48:13.148390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.148368    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.154011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.153986    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-894bbb-dockercfg-g9rdb\""
Nov 02 07:48:13.162749 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.162723    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmf6\" (UniqueName: \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.162986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.162966    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.163247 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.163229    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.163443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.163427    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.163580 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.163566    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.163696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.163682    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmt6k\" (UniqueName: \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.164756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.164735    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:48:13.264757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.264716    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.264919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.264776    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.264974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.264922    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.265031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.264983    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.265084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265027    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt6k\" (UniqueName: \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.265139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265097    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlpn\" (UniqueName: \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.265206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265179    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmf6\" (UniqueName: \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.265293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265268    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.265678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265656    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.265765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.265714    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.267660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.267642    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.267886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.267864    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.352254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.316925    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt6k\" (UniqueName: \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\") pod \"payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.378337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.355008    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmf6\" (UniqueName: \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\") pod \"maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.378337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.366570    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.378337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.366802    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlpn\" (UniqueName: \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.378337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.369019    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.378337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.369345    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:13.424489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.415794    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlpn\" (UniqueName: \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\") pod \"payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.487411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.467583    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:13.654978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.654785    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:13.832018 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:13.831989    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2351e5_affe_4450_8886_258cccc65413.slice/crio-fc22f4ea7b348412c12e6a4295f4316f02beddde9e60835224156db12981d42a WatchSource:0}: Error finding container fc22f4ea7b348412c12e6a4295f4316f02beddde9e60835224156db12981d42a: Status 404 returned error can't find the container with id fc22f4ea7b348412c12e6a4295f4316f02beddde9e60835224156db12981d42a
Nov 02 07:48:13.832873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.832852    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:48:13.849001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.848972    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:48:13.853627 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:13.853569    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a004714_7dd3_40d4_95c6_769fc5e50e19.slice/crio-f6967ad85b88346c5a023839f52c2af8eac5dd834a9660c84943cba91899c22f WatchSource:0}: Error finding container f6967ad85b88346c5a023839f52c2af8eac5dd834a9660c84943cba91899c22f: Status 404 returned error can't find the container with id f6967ad85b88346c5a023839f52c2af8eac5dd834a9660c84943cba91899c22f
Nov 02 07:48:13.990921 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:13.990881    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:48:13.995243 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:13.995213    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6378e1c5_9db9_44ba_bb05_304c910a1b61.slice/crio-5cce0eb739dbb8e42f62fef3b786d590a92b6c497abc11e1eea72f1d85534b36 WatchSource:0}: Error finding container 5cce0eb739dbb8e42f62fef3b786d590a92b6c497abc11e1eea72f1d85534b36: Status 404 returned error can't find the container with id 5cce0eb739dbb8e42f62fef3b786d590a92b6c497abc11e1eea72f1d85534b36
Nov 02 07:48:14.727179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.727084    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" event={"ID":"6378e1c5-9db9-44ba-bb05-304c910a1b61","Type":"ContainerStarted","Data":"195edf8279ad356174c2df6b4dede79661a7a65a2935123fae1a8dd29ce97428"}
Nov 02 07:48:14.727179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.727164    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:14.728287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.727191    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" event={"ID":"6378e1c5-9db9-44ba-bb05-304c910a1b61","Type":"ContainerStarted","Data":"5cce0eb739dbb8e42f62fef3b786d590a92b6c497abc11e1eea72f1d85534b36"}
Nov 02 07:48:14.728987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.728960    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" event={"ID":"9a004714-7dd3-40d4-95c6-769fc5e50e19","Type":"ContainerStarted","Data":"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"}
Nov 02 07:48:14.729089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.729019    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:14.729089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.729043    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" event={"ID":"9a004714-7dd3-40d4-95c6-769fc5e50e19","Type":"ContainerStarted","Data":"f6967ad85b88346c5a023839f52c2af8eac5dd834a9660c84943cba91899c22f"}
Nov 02 07:48:14.730733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.730707    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" event={"ID":"da2351e5-affe-4450-8886-258cccc65413","Type":"ContainerStarted","Data":"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"}
Nov 02 07:48:14.730828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.730756    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" event={"ID":"da2351e5-affe-4450-8886-258cccc65413","Type":"ContainerStarted","Data":"fc22f4ea7b348412c12e6a4295f4316f02beddde9e60835224156db12981d42a"}
Nov 02 07:48:14.730828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.730789    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:14.776200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.776142    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" podStartSLOduration=1.451318187 podStartE2EDuration="1.776127196s" totalImagesPullingTime="324.809009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:14.772825682 +0000 UTC m=+1439.535262589" watchObservedRunningTime="2026-11-02 07:48:14.776127196 +0000 UTC m=+1439.538564101"
Nov 02 07:48:14.815911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.815853    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" podStartSLOduration=1.815838894 podStartE2EDuration="1.815838894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:14.813490217 +0000 UTC m=+1439.575927128" watchObservedRunningTime="2026-11-02 07:48:14.815838894 +0000 UTC m=+1439.578275799"
Nov 02 07:48:14.864506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:14.864436    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" podStartSLOduration=1.8644178409999999 podStartE2EDuration="1.864417841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:14.862899294 +0000 UTC m=+1439.625336202" watchObservedRunningTime="2026-11-02 07:48:14.864417841 +0000 UTC m=+1439.626854748"
Nov 02 07:48:17.724649 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:17.724610    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:48:17.751465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.751430    4515 generic.go:356] "Generic (PLEG): container finished" podID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerID="88f39b68defefd59593caaa0021922e29e7207f28f1f71f5da8a5bba8c77113e" exitCode=0
Nov 02 07:48:17.751640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.751491    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerDied","Data":"88f39b68defefd59593caaa0021922e29e7207f28f1f71f5da8a5bba8c77113e"}
Nov 02 07:48:17.751949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.751932    4515 scope.go:122] "RemoveContainer" containerID="88f39b68defefd59593caaa0021922e29e7207f28f1f71f5da8a5bba8c77113e"
Nov 02 07:48:17.753473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.753443    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:17.753541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.753515    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:17.753584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:17.753546    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:18.760701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:18.760657    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerStarted","Data":"808c39ccb8fab7e9c86b7c0b429f2322dd86339cfcfb699b3ed6a79199e110e1"}
Nov 02 07:48:19.408478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:19.408444    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:48:19.409728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:19.409701    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 07:48:19.409851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:19.409752    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 07:48:20.412333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:20.412040    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 07:48:20.412333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:20.412108    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 07:48:20.747100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:20.747029    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:48:21.408895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:21.408859    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 07:48:21.409067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:21.408915    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 07:48:21.841486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:21.841448    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:48:22.408689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:22.408654    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 07:48:22.408876 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:22.408706    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 07:48:23.409082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:23.409049    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 07:48:23.409526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:23.409100    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 07:48:24.447109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.444822    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:48:24.447630 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.447406    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:48:24.923679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.923587    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:48:24.932955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.932920    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:24.975857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.974917    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:48:24.982234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.982201    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f3716\""
Nov 02 07:48:24.982447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:24.982286    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f3716-dockercfg-d4vlh\""
Nov 02 07:48:25.038430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.038374    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.038635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.038521    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbgs\" (UniqueName: \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.139966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.139917    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbgs\" (UniqueName: \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.140185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.140113    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.140866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.140841    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.163544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.163513    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbgs\" (UniqueName: \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\") pod \"payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.281112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.281080    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.613166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.613119    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:48:25.613875 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:25.613843    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172a4882_20f2_4f7a_a34b_b35edee77fd5.slice/crio-96e633b340ab956bcf6a35b25978f7fd646513a7d19d28572bbdc976e398723e WatchSource:0}: Error finding container 96e633b340ab956bcf6a35b25978f7fd646513a7d19d28572bbdc976e398723e: Status 404 returned error can't find the container with id 96e633b340ab956bcf6a35b25978f7fd646513a7d19d28572bbdc976e398723e
Nov 02 07:48:25.740577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.740545    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:48:25.740800 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.740787    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:48:25.871606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.871511    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" event={"ID":"172a4882-20f2-4f7a-a34b-b35edee77fd5","Type":"ContainerStarted","Data":"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"}
Nov 02 07:48:25.871606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.871588    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:25.871833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.871618    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" event={"ID":"172a4882-20f2-4f7a-a34b-b35edee77fd5","Type":"ContainerStarted","Data":"96e633b340ab956bcf6a35b25978f7fd646513a7d19d28572bbdc976e398723e"}
Nov 02 07:48:25.913956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:25.913907    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" podStartSLOduration=1.913893366 podStartE2EDuration="1.913893366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:25.912061651 +0000 UTC m=+1450.674498560" watchObservedRunningTime="2026-11-02 07:48:25.913893366 +0000 UTC m=+1450.676330271"
Nov 02 07:48:26.644991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:26.644956    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:48:32.971496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:32.971419    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:48:33.106700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.106670    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:48:33.107153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.107132    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.115283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.114905    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.115283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.115007    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tx58\" (UniqueName: \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.147758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.147720    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b2f76f\""
Nov 02 07:48:33.185448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.183778    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b2f76f-dockercfg-qngg9\""
Nov 02 07:48:33.185448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.184849    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:48:33.210036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.209997    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:48:33.210368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.210301    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.216230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.216202    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.216398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.216302    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.216472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.216418    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tx58\" (UniqueName: \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.216535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.216481    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmqb\" (UniqueName: \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.230392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.217484    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.270961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.270031    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tx58\" (UniqueName: \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\") pod \"payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.339335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.321969    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmqb\" (UniqueName: \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.339335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.322117    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.339335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.323047    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.395520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.395491    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmqb\" (UniqueName: \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\") pod \"payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.469626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.469588    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:33.590336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.583438    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:33.990006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:33.989843    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:48:33.991106 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:33.991052    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c29bf_a3c2_49f1_ad48_2b2579caf7cd.slice/crio-0db9dbf9a7da00885041a778d2ad044cad559863b0c43678d07338e8ec740398 WatchSource:0}: Error finding container 0db9dbf9a7da00885041a778d2ad044cad559863b0c43678d07338e8ec740398: Status 404 returned error can't find the container with id 0db9dbf9a7da00885041a778d2ad044cad559863b0c43678d07338e8ec740398
Nov 02 07:48:34.126091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.126037    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:48:34.137762 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:34.137727    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43299bc1_855b_40e1_a5d0_76f61a739f50.slice/crio-69dc3afee3fa049cc47cac26f25438cd22263ef32637ff8c4fe1b0955194d242 WatchSource:0}: Error finding container 69dc3afee3fa049cc47cac26f25438cd22263ef32637ff8c4fe1b0955194d242: Status 404 returned error can't find the container with id 69dc3afee3fa049cc47cac26f25438cd22263ef32637ff8c4fe1b0955194d242
Nov 02 07:48:34.144937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.144909    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" event={"ID":"160c29bf-a3c2-49f1-ad48-2b2579caf7cd","Type":"ContainerStarted","Data":"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"}
Nov 02 07:48:34.145056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.144970    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" event={"ID":"160c29bf-a3c2-49f1-ad48-2b2579caf7cd","Type":"ContainerStarted","Data":"0db9dbf9a7da00885041a778d2ad044cad559863b0c43678d07338e8ec740398"}
Nov 02 07:48:34.145259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.145204    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:34.187089 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.187033    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" podStartSLOduration=2.187019442 podStartE2EDuration="2.187019442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:34.185865471 +0000 UTC m=+1458.948302379" watchObservedRunningTime="2026-11-02 07:48:34.187019442 +0000 UTC m=+1458.949456347"
Nov 02 07:48:34.537877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.537810    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:48:34.551867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.551834    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.574395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.574276    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-894bbb-serving-cert\""
Nov 02 07:48:34.590123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.589770    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.590123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.589874    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.590123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.589972    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.590439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.590208    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrfv\" (UniqueName: \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.595654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.595608    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:48:34.695718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.695679    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrfv\" (UniqueName: \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.695917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.695801    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.695917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.695896    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.696748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.695991    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.696748 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:34.696284    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-serving-cert: secret "maas-api-e2e-praxis-a-894bbb-serving-cert" not found
Nov 02 07:48:34.696748 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:34.696369    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls podName:b80f0544-158e-4eff-b810-f80803306c73 nodeName:}" failed. No retries permitted until 2026-11-02 07:48:35.196346367 +0000 UTC m=+1459.958783264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls") pod "maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" (UID: "b80f0544-158e-4eff-b810-f80803306c73") : secret "maas-api-e2e-praxis-a-894bbb-serving-cert" not found
Nov 02 07:48:34.697046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.697021    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.699172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.699145    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:34.722444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:34.722416    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrfv\" (UniqueName: \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:35.153370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.153295    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" event={"ID":"43299bc1-855b-40e1-a5d0-76f61a739f50","Type":"ContainerStarted","Data":"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"}
Nov 02 07:48:35.153763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.153622    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:35.153763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.153661    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" event={"ID":"43299bc1-855b-40e1-a5d0-76f61a739f50","Type":"ContainerStarted","Data":"69dc3afee3fa049cc47cac26f25438cd22263ef32637ff8c4fe1b0955194d242"}
Nov 02 07:48:35.201236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.201174    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" podStartSLOduration=3.201155939 podStartE2EDuration="3.201155939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:35.197538726 +0000 UTC m=+1459.959975633" watchObservedRunningTime="2026-11-02 07:48:35.201155939 +0000 UTC m=+1459.963592840"
Nov 02 07:48:35.209774 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.209730    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:35.212584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.212554    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:35.474911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.474809    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:35.895097 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:35.895060    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80f0544_158e_4eff_b810_f80803306c73.slice/crio-f87b16d63d579c0d1c1475162198afb8e8e19dade6b91c9f2c6c225d6692fe20 WatchSource:0}: Error finding container f87b16d63d579c0d1c1475162198afb8e8e19dade6b91c9f2c6c225d6692fe20: Status 404 returned error can't find the container with id f87b16d63d579c0d1c1475162198afb8e8e19dade6b91c9f2c6c225d6692fe20
Nov 02 07:48:35.906091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:35.906034    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:48:36.169397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:36.165679    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" event={"ID":"b80f0544-158e-4eff-b810-f80803306c73","Type":"ContainerStarted","Data":"f87b16d63d579c0d1c1475162198afb8e8e19dade6b91c9f2c6c225d6692fe20"}
Nov 02 07:48:36.879213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:36.879156    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:48:37.176658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:37.176565    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" event={"ID":"b80f0544-158e-4eff-b810-f80803306c73","Type":"ContainerStarted","Data":"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"}
Nov 02 07:48:37.177045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:37.176673    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:37.224637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:37.224574    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" podStartSLOduration=2.792487564 podStartE2EDuration="3.224555084s" totalImagesPullingTime="432.06752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:37.22070579 +0000 UTC m=+1461.983142695" watchObservedRunningTime="2026-11-02 07:48:37.224555084 +0000 UTC m=+1461.986991991"
Nov 02 07:48:37.272483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:37.272446    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:48:37.272696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:37.272505    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:48:38.112524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:38.112480    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:48:38.112720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:38.112570    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:48:38.900660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:38.876918    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:48:39.638771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:38.915813    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:48:40.200658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.200627    4515 generic.go:356] "Generic (PLEG): container finished" podID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerID="8007e6257ffbe6a24795c5b49de232c518c443099956da4185af4d66d2d7ad7c" exitCode=0
Nov 02 07:48:40.201172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.200700    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerDied","Data":"8007e6257ffbe6a24795c5b49de232c518c443099956da4185af4d66d2d7ad7c"}
Nov 02 07:48:40.201231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.201193    4515 scope.go:122] "RemoveContainer" containerID="8007e6257ffbe6a24795c5b49de232c518c443099956da4185af4d66d2d7ad7c"
Nov 02 07:48:40.202556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.202534    4515 generic.go:356] "Generic (PLEG): container finished" podID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerID="463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4" exitCode=0
Nov 02 07:48:40.202657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.202617    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerDied","Data":"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"}
Nov 02 07:48:40.202942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.202910    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.203017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.202992    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.203017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.203003    4515 scope.go:122] "RemoveContainer" containerID="463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"
Nov 02 07:48:40.203128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.203044    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.204409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204386    4515 generic.go:356] "Generic (PLEG): container finished" podID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerID="f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e" exitCode=0
Nov 02 07:48:40.204507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204411    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerDied","Data":"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"}
Nov 02 07:48:40.204703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204676    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.204744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204735    4515 scope.go:122] "RemoveContainer" containerID="f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"
Nov 02 07:48:40.204776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204752    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.204815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.204800    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.206593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.206569    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.206651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.206636    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:40.206697 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:40.206682    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:41.258331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:41.253372    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerStarted","Data":"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"}
Nov 02 07:48:41.275333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:41.274528    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerStarted","Data":"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"}
Nov 02 07:48:41.306477 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:41.306387    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerStarted","Data":"23abdf782b9e0320103f5ecdb73cf5149a8aab43d9f131a4caaae18b657dbcb6"}
Nov 02 07:48:42.143354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.140248    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:48:42.143354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.140432    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:42.143354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.140502    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:42.205334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.198616    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:48:42.205334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.198760    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:42.205334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.198803    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:42.302387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.302355    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:48:42.302901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.302883    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:42.303013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.302990    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:42.873335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.854516    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:48:42.874001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:42.873938    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" podUID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" containerName="maas-api" containerID="cri-o://bef1e5d46264f8b04501574d699723b38680990d85f909f0b7657ed9c5fa4df2" gracePeriod=30
Nov 02 07:48:43.165414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.142652    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:43.165414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.142732    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:43.209334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.205886    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:43.209334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.205946    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:43.254336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.251291    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:48:43.362335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.361594    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:43.362335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.361662    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:43.668330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.657600    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:48:43.768732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.754641    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:48:43.802074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.796346    4515 generic.go:356] "Generic (PLEG): container finished" podID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" containerID="bef1e5d46264f8b04501574d699723b38680990d85f909f0b7657ed9c5fa4df2" exitCode=0
Nov 02 07:48:43.802074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.796476    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" event={"ID":"86a4b60b-187a-4a6b-8d3e-3de6eee097ff","Type":"ContainerDied","Data":"bef1e5d46264f8b04501574d699723b38680990d85f909f0b7657ed9c5fa4df2"}
Nov 02 07:48:43.885681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.884864    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" podUID="a21ba385-c10d-4c41-8093-b8d2910c593e" containerName="payload-processing" containerID="cri-o://cf8b21952d2a8993a71da8cf28601fb272389ec29510d0990d162aee9072c14d" gracePeriod=25
Nov 02 07:48:43.916121 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:43.916063    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" podUID="45c7d94b-b221-43ef-a22c-baaa92896282" containerName="payload-pre-processing" containerID="cri-o://41225ca38ab9bc32e7503d4239a082d18d948f92086653c4eadbbb33869dd22e" gracePeriod=25
Nov 02 07:48:44.158529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.157561    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:44.158529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.157638    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:44.178171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.177070    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:48:44.209252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.209157    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:44.210441 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.210291    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:44.532334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.532279    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:44.533031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.532993    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:44.727335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.721530    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.906413    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\") pod \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") "
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.906467    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\") pod \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") "
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.906774    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\" (UniqueName: \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\") pod \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") "
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.906826    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\") pod \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\" (UID: \"86a4b60b-187a-4a6b-8d3e-3de6eee097ff\") "
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.908675    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca" pod "86a4b60b-187a-4a6b-8d3e-3de6eee097ff" (UID: "86a4b60b-187a-4a6b-8d3e-3de6eee097ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.915043    4515 generic.go:356] "Generic (PLEG): container finished" podID="a21ba385-c10d-4c41-8093-b8d2910c593e" containerID="cf8b21952d2a8993a71da8cf28601fb272389ec29510d0990d162aee9072c14d" exitCode=0
Nov 02 07:48:44.915341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.915283    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" event={"ID":"a21ba385-c10d-4c41-8093-b8d2910c593e","Type":"ContainerDied","Data":"cf8b21952d2a8993a71da8cf28601fb272389ec29510d0990d162aee9072c14d"}
Nov 02 07:48:44.926339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.924561    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn" event={"ID":"86a4b60b-187a-4a6b-8d3e-3de6eee097ff","Type":"ContainerDied","Data":"40f5c51dac626ddda240ae42be7a16d1808a7adcd89ead81a3ce83e9796b8261"}
Nov 02 07:48:44.941254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.926600    4515 scope.go:122] "RemoveContainer" containerID="bef1e5d46264f8b04501574d699723b38680990d85f909f0b7657ed9c5fa4df2"
Nov 02 07:48:44.941254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.926802    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"
Nov 02 07:48:44.964226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.964189    4515 generic.go:356] "Generic (PLEG): container finished" podID="45c7d94b-b221-43ef-a22c-baaa92896282" containerID="41225ca38ab9bc32e7503d4239a082d18d948f92086653c4eadbbb33869dd22e" exitCode=0
Nov 02 07:48:44.964485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:44.964463    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" event={"ID":"45c7d94b-b221-43ef-a22c-baaa92896282","Type":"ContainerDied","Data":"41225ca38ab9bc32e7503d4239a082d18d948f92086653c4eadbbb33869dd22e"}
Nov 02 07:48:45.011962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.009603    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:45.024182 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.024141    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs" pod "86a4b60b-187a-4a6b-8d3e-3de6eee097ff" (UID: "86a4b60b-187a-4a6b-8d3e-3de6eee097ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:48:45.059591 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.059505    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf" pod "86a4b60b-187a-4a6b-8d3e-3de6eee097ff" (UID: "86a4b60b-187a-4a6b-8d3e-3de6eee097ff"). InnerVolumeSpecName "kube-api-access-glsvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:45.065884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.065850    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls" pod "86a4b60b-187a-4a6b-8d3e-3de6eee097ff" (UID: "86a4b60b-187a-4a6b-8d3e-3de6eee097ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:48:45.131339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.116619    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glsvf\" (UniqueName: \"kubernetes.io/projected/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-kube-api-access-glsvf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:45.131339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.116663    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:45.131339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.116684    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a4b60b-187a-4a6b-8d3e-3de6eee097ff-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:45.154111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.153701    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:45.154111 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.153756    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:45.191357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.191302    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:45.198277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.198241    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:45.198560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.191788    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:48:45.303129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.302688    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:45.303129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.302749    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:45.515174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.515136    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:48:45.663775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.615137    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7fde8-657fcf64f5-98svn"]
Nov 02 07:48:45.804197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.680633    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:48:45.813450 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:45.813427    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:48:46.002923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.002880    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\" (UniqueName: \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\") pod \"45c7d94b-b221-43ef-a22c-baaa92896282\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") "
Nov 02 07:48:46.003062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.002949    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\") pod \"45c7d94b-b221-43ef-a22c-baaa92896282\" (UID: \"45c7d94b-b221-43ef-a22c-baaa92896282\") "
Nov 02 07:48:46.003062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.003039    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\" (UniqueName: \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\") pod \"a21ba385-c10d-4c41-8093-b8d2910c593e\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") "
Nov 02 07:48:46.003193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.003130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\") pod \"a21ba385-c10d-4c41-8093-b8d2910c593e\" (UID: \"a21ba385-c10d-4c41-8093-b8d2910c593e\") "
Nov 02 07:48:46.012915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.012878    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume" pod "45c7d94b-b221-43ef-a22c-baaa92896282" (UID: "45c7d94b-b221-43ef-a22c-baaa92896282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:46.064337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.061774    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22" pod "a21ba385-c10d-4c41-8093-b8d2910c593e" (UID: "a21ba385-c10d-4c41-8093-b8d2910c593e"). InnerVolumeSpecName "kube-api-access-57t22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:46.069331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.067682    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w" pod "45c7d94b-b221-43ef-a22c-baaa92896282" (UID: "45c7d94b-b221-43ef-a22c-baaa92896282"). InnerVolumeSpecName "kube-api-access-gnc5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:46.109530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.109436    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57t22\" (UniqueName: \"kubernetes.io/projected/a21ba385-c10d-4c41-8093-b8d2910c593e-kube-api-access-57t22\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:46.109530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.109470    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnc5w\" (UniqueName: \"kubernetes.io/projected/45c7d94b-b221-43ef-a22c-baaa92896282-kube-api-access-gnc5w\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:46.109530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.109486    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c7d94b-b221-43ef-a22c-baaa92896282-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:46.110537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.110500    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume" pod "a21ba385-c10d-4c41-8093-b8d2910c593e" (UID: "a21ba385-c10d-4c41-8093-b8d2910c593e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:46.197091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.196515    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:46.197091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.196616    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:46.197091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.196763    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:46.197091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.196808    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:46.197997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.197972    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:48:46.203543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.203520    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"
Nov 02 07:48:46.204483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.203527    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d" event={"ID":"a21ba385-c10d-4c41-8093-b8d2910c593e","Type":"ContainerDied","Data":"8c00945c5818667b855535322e87a3f51fd5cd46012b4f1ddeba222a89c0f675"}
Nov 02 07:48:46.204483 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.204234    4515 scope.go:122] "RemoveContainer" containerID="cf8b21952d2a8993a71da8cf28601fb272389ec29510d0990d162aee9072c14d"
Nov 02 07:48:46.207503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.207432    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh" event={"ID":"45c7d94b-b221-43ef-a22c-baaa92896282","Type":"ContainerDied","Data":"818730a5f1ea88423e9ded14f727af8fe3bfd397a65069d731f0f92a44df166d"}
Nov 02 07:48:46.210071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.207877    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"
Nov 02 07:48:46.211001 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.210979    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a21ba385-c10d-4c41-8093-b8d2910c593e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:46.309624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.305891    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:46.309624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.305940    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:46.309624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.308654    4515 scope.go:122] "RemoveContainer" containerID="41225ca38ab9bc32e7503d4239a082d18d948f92086653c4eadbbb33869dd22e"
Nov 02 07:48:46.409691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.409661    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:48:46.505627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.505590    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7fde8-56fdc75bc5-nnc4d"]
Nov 02 07:48:46.511797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.511732    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:48:46.572421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:46.565797    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7fde8-5b8cc64cd7-d9lhh"]
Nov 02 07:48:47.139828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.139790    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 02 07:48:47.140487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.140441    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 02 07:48:47.191926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.191891    4515 patch_prober.go:29] interesting pod/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 02 07:48:47.192294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.192262    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 02 07:48:47.304752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.304708    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:48:47.305059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.305026    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 07:48:47.402686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.402603    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c7d94b-b221-43ef-a22c-baaa92896282" path="/var/lib/kubelet/pods/45c7d94b-b221-43ef-a22c-baaa92896282/volumes"
Nov 02 07:48:47.404838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.404812    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" path="/var/lib/kubelet/pods/86a4b60b-187a-4a6b-8d3e-3de6eee097ff/volumes"
Nov 02 07:48:47.405629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:47.405606    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21ba385-c10d-4c41-8093-b8d2910c593e" path="/var/lib/kubelet/pods/a21ba385-c10d-4c41-8093-b8d2910c593e/volumes"
Nov 02 07:48:47.796551 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:47.796511    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:48:48.385361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.385303    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:48:48.386383 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.386359    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:48:48.387374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.387350    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:48:48.389157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.389133    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:48:48.390162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.390143    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:48:48.398611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.398585    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:48:48.399260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.399239    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:48:48.400161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.400134    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:48:48.658538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.658453    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" podUID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" containerName="payload-processing" containerID="cri-o://98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1" gracePeriod=25
Nov 02 07:48:48.743016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.742981    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:48:48.743348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.743287    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy" containerID="cri-o://75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172" gracePeriod=30
Nov 02 07:48:48.755490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:48.755404    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" podUID="4c9664ac-03b6-4e78-956d-689790d5422a" containerName="payload-pre-processing" containerID="cri-o://e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3" gracePeriod=25
Nov 02 07:48:49.124719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.124686    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:48:49.216015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.215984    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:48:49.276004 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.275966    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\") pod \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") "
Nov 02 07:48:49.276243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.276027    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\" (UniqueName: \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\") pod \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\" (UID: \"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48\") "
Nov 02 07:48:49.292934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.292150    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume" pod "b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" (UID: "b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:49.300532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.300492    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr" pod "b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" (UID: "b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48"). InnerVolumeSpecName "kube-api-access-k54nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:49.308720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.308692    4515 generic.go:356] "Generic (PLEG): container finished" podID="4c9664ac-03b6-4e78-956d-689790d5422a" containerID="e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3" exitCode=0
Nov 02 07:48:49.309028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.308743    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"
Nov 02 07:48:49.309372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.308765    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" event={"ID":"4c9664ac-03b6-4e78-956d-689790d5422a","Type":"ContainerDied","Data":"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"}
Nov 02 07:48:49.309493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.309423    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr" event={"ID":"4c9664ac-03b6-4e78-956d-689790d5422a","Type":"ContainerDied","Data":"d3785f623fb22fa7eb4730b85548f799d6c08993c925b5ba8b6e08f246399a97"}
Nov 02 07:48:49.309493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.309459    4515 scope.go:122] "RemoveContainer" containerID="e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"
Nov 02 07:48:49.311970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.311947    4515 generic.go:356] "Generic (PLEG): container finished" podID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" containerID="98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1" exitCode=0
Nov 02 07:48:49.312115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.311996    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"
Nov 02 07:48:49.312177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.312029    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" event={"ID":"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48","Type":"ContainerDied","Data":"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"}
Nov 02 07:48:49.312177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.312149    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k" event={"ID":"b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48","Type":"ContainerDied","Data":"d23fa1de251be031f91883c243e92e7a7dd410a8cc6f39bd6d3b9e5f6d456612"}
Nov 02 07:48:49.323576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.323556    4515 scope.go:122] "RemoveContainer" containerID="e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"
Nov 02 07:48:49.323957 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:49.323927    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3\": container with ID starting with e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3 not found: ID does not exist" containerID="e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"
Nov 02 07:48:49.324066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.323966    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3"} err="failed to get container status \"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3\": rpc error: code = NotFound desc = could not find container \"e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3\": container with ID starting with e0bee2204465100c891ec36f3a82cbe439cc20b6f4ccadf119d8b3d9daf977e3 not found: ID does not exist"
Nov 02 07:48:49.324066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.323992    4515 scope.go:122] "RemoveContainer" containerID="98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"
Nov 02 07:48:49.337973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.337858    4515 scope.go:122] "RemoveContainer" containerID="98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"
Nov 02 07:48:49.338597 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:49.338561    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1\": container with ID starting with 98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1 not found: ID does not exist" containerID="98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"
Nov 02 07:48:49.338722 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.338604    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1"} err="failed to get container status \"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1\": rpc error: code = NotFound desc = could not find container \"98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1\": container with ID starting with 98f1de8145960e34b3f75c7580c7d5bbe686351020c8c22ce16c438bd9c6f4e1 not found: ID does not exist"
Nov 02 07:48:49.363418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.363301    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:48:49.375972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.375949    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-b48767f89-cb28k"]
Nov 02 07:48:49.377280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377254    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\") pod \"4c9664ac-03b6-4e78-956d-689790d5422a\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") "
Nov 02 07:48:49.377422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377393    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\" (UniqueName: \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\") pod \"4c9664ac-03b6-4e78-956d-689790d5422a\" (UID: \"4c9664ac-03b6-4e78-956d-689790d5422a\") "
Nov 02 07:48:49.377611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377591    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume" pod "4c9664ac-03b6-4e78-956d-689790d5422a" (UID: "4c9664ac-03b6-4e78-956d-689790d5422a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:49.377945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377917    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9664ac-03b6-4e78-956d-689790d5422a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:49.377945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377940    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:49.378058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.377955    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54nr\" (UniqueName: \"kubernetes.io/projected/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48-kube-api-access-k54nr\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:49.379449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.379424    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r" pod "4c9664ac-03b6-4e78-956d-689790d5422a" (UID: "4c9664ac-03b6-4e78-956d-689790d5422a"). InnerVolumeSpecName "kube-api-access-grq8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:49.479387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.479339    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grq8r\" (UniqueName: \"kubernetes.io/projected/4c9664ac-03b6-4e78-956d-689790d5422a-kube-api-access-grq8r\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:49.661144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.661104    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:48:49.664813 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:49.664786    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-8688bfc8dc-f9rdr"]
Nov 02 07:48:51.374165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:51.374128    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9664ac-03b6-4e78-956d-689790d5422a" path="/var/lib/kubelet/pods/4c9664ac-03b6-4e78-956d-689790d5422a/volumes"
Nov 02 07:48:51.374609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:51.374594    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" path="/var/lib/kubelet/pods/b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48/volumes"
Nov 02 07:48:54.062616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.062590    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:48:54.246034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246002    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246104    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246137    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246166    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246198    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246234    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246276    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246451    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246479    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:54.246538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246500    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:54.246693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246537    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\") pod \"e1468920-9ced-4001-a0e4-4721f031c853\" (UID: \"e1468920-9ced-4001-a0e4-4721f031c853\") "
Nov 02 07:48:54.246693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246580    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:54.246808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246699    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:54.246922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.246903    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:54.247213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.247191    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.247272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.247221    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.247272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.247235    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.247272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.247248    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.247272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.247261    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1468920-9ced-4001-a0e4-4721f031c853-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.248658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.248630    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:54.248658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.248635    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "kube-api-access-khsnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:54.248924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.248903    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:54.248973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.248928    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token" pod "e1468920-9ced-4001-a0e4-4721f031c853" (UID: "e1468920-9ced-4001-a0e4-4721f031c853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:54.348167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.348071    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1468920-9ced-4001-a0e4-4721f031c853-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.348167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.348101    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khsnk\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-kube-api-access-khsnk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.348167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.348117    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1468920-9ced-4001-a0e4-4721f031c853-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.348167 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.348128    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1468920-9ced-4001-a0e4-4721f031c853-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:48:54.349905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.349877    4515 generic.go:356] "Generic (PLEG): container finished" podID="e1468920-9ced-4001-a0e4-4721f031c853" containerID="75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172" exitCode=0
Nov 02 07:48:54.350037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.349941    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"
Nov 02 07:48:54.350037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.349956    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerDied","Data":"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"}
Nov 02 07:48:54.350037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.350032    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c" event={"ID":"e1468920-9ced-4001-a0e4-4721f031c853","Type":"ContainerDied","Data":"0bafe696106df2772fad4b1d3fabf3750bd337411f1e84e349a32a2688653a4d"}
Nov 02 07:48:54.350175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.350056    4515 scope.go:122] "RemoveContainer" containerID="75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"
Nov 02 07:48:54.360668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.360648    4515 scope.go:122] "RemoveContainer" containerID="12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"
Nov 02 07:48:54.369983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.369964    4515 scope.go:122] "RemoveContainer" containerID="75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"
Nov 02 07:48:54.370244 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:54.370226    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172\": container with ID starting with 75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172 not found: ID does not exist" containerID="75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"
Nov 02 07:48:54.370288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.370252    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172"} err="failed to get container status \"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172\": rpc error: code = NotFound desc = could not find container \"75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172\": container with ID starting with 75e1a11509216a8363a53dd49620ed5d3982df021edd6e3dd49f5d320ae49172 not found: ID does not exist"
Nov 02 07:48:54.370288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.370271    4515 scope.go:122] "RemoveContainer" containerID="12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"
Nov 02 07:48:54.370541 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:48:54.370523    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096\": container with ID starting with 12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096 not found: ID does not exist" containerID="12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"
Nov 02 07:48:54.370597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.370550    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096"} err="failed to get container status \"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096\": rpc error: code = NotFound desc = could not find container \"12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096\": container with ID starting with 12fefbe402beaaa9decd25202fa864a0f2006cef2e8abdfc5ba9584ba9380096 not found: ID does not exist"
Nov 02 07:48:54.410601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.410566    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:48:54.423401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:54.423375    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7fde8-openshift-default-764f58bbb5-lwp2c"]
Nov 02 07:48:55.205140 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.205107    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:48:55.240809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.240777    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:48:55.374755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.374724    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1468920-9ced-4001-a0e4-4721f031c853" path="/var/lib/kubelet/pods/e1468920-9ced-4001-a0e4-4721f031c853/volumes"
Nov 02 07:48:55.546207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546176    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:48:55.546833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546818    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" containerName="payload-processing"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546834    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" containerName="payload-processing"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546848    4515 cpu_manager.go:414] "containerMap: removing container" podUID="a21ba385-c10d-4c41-8093-b8d2910c593e" containerName="payload-processing"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546855    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21ba385-c10d-4c41-8093-b8d2910c593e" containerName="payload-processing"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546866    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546872    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:48:55.546896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546894    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4c9664ac-03b6-4e78-956d-689790d5422a" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546899    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9664ac-03b6-4e78-956d-689790d5422a" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546907    4515 cpu_manager.go:414] "containerMap: removing container" podUID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" containerName="maas-api"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546912    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" containerName="maas-api"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546923    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546929    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546940    4515 cpu_manager.go:414] "containerMap: removing container" podUID="45c7d94b-b221-43ef-a22c-baaa92896282" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.546945    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c7d94b-b221-43ef-a22c-baaa92896282" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547027    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547035    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c7d94b-b221-43ef-a22c-baaa92896282" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547045    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b5a74a-d8c4-4c87-8384-ecd7ce5cba48" containerName="payload-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547052    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21ba385-c10d-4c41-8093-b8d2910c593e" containerName="payload-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547063    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9664ac-03b6-4e78-956d-689790d5422a" containerName="payload-pre-processing"
Nov 02 07:48:55.547086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.547068    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a4b60b-187a-4a6b-8d3e-3de6eee097ff" containerName="maas-api"
Nov 02 07:48:55.552186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.552165    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.569603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.569579    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b2f76f-serving-cert\""
Nov 02 07:48:55.574276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.574251    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:48:55.688788 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.688754    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.688990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.688973    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.689156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.689127    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.689220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.689188    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkchp\" (UniqueName: \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.790037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.789996    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.790037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.790039    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkchp\" (UniqueName: \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.790294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.790104    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.790294 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.790274    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.790792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.790773    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.792641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.792612    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.792755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.792646    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.813954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.813886    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkchp\" (UniqueName: \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\") pod \"maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:55.869332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:55.866300    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:56.376342 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:48:56.376251    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cc7094_c2c8_4a6b_b8c4_a0cd868a5111.slice/crio-a7287823935fa6b44e51fbdeb52412e98c0e1954269b66efe39f2a4dd7d7ba5d WatchSource:0}: Error finding container a7287823935fa6b44e51fbdeb52412e98c0e1954269b66efe39f2a4dd7d7ba5d: Status 404 returned error can't find the container with id a7287823935fa6b44e51fbdeb52412e98c0e1954269b66efe39f2a4dd7d7ba5d
Nov 02 07:48:56.376904 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:56.376837    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:48:57.378558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:57.378520    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" event={"ID":"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111","Type":"ContainerStarted","Data":"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"}
Nov 02 07:48:57.378985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:57.378587    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:48:57.378985 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:57.378612    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" event={"ID":"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111","Type":"ContainerStarted","Data":"a7287823935fa6b44e51fbdeb52412e98c0e1954269b66efe39f2a4dd7d7ba5d"}
Nov 02 07:48:57.424090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:57.424039    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" podStartSLOduration=2.069786094 podStartE2EDuration="2.424027286s" totalImagesPullingTime="354.241192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:57.421641689 +0000 UTC m=+1482.184078595" watchObservedRunningTime="2026-11-02 07:48:57.424027286 +0000 UTC m=+1482.186464216"
Nov 02 07:48:58.517022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:48:58.513708    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:49:00.206382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.206285    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" podUID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" containerName="payload-processing" containerID="cri-o://e3fd50398234caff08de646f26e7ce9c04b1e93c756d0cbc247e37a3d0f4a762" gracePeriod=25
Nov 02 07:49:00.243331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.242385    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" podUID="80e054c4-97a5-447a-a16f-15a004745833" containerName="payload-pre-processing" containerID="cri-o://b5d2ea929df80cc6c5b9bc351992885683535c7ce2cd93fc7c6c97d620485ac1" gracePeriod=25
Nov 02 07:49:00.459449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.459356    4515 generic.go:356] "Generic (PLEG): container finished" podID="80e054c4-97a5-447a-a16f-15a004745833" containerID="b5d2ea929df80cc6c5b9bc351992885683535c7ce2cd93fc7c6c97d620485ac1" exitCode=0
Nov 02 07:49:00.459612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.459515    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" event={"ID":"80e054c4-97a5-447a-a16f-15a004745833","Type":"ContainerDied","Data":"b5d2ea929df80cc6c5b9bc351992885683535c7ce2cd93fc7c6c97d620485ac1"}
Nov 02 07:49:00.461805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.461774    4515 generic.go:356] "Generic (PLEG): container finished" podID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" containerID="e3fd50398234caff08de646f26e7ce9c04b1e93c756d0cbc247e37a3d0f4a762" exitCode=0
Nov 02 07:49:00.461969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.461949    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" event={"ID":"ba8fd132-3171-4e4e-b1ed-5ff515e83621","Type":"ContainerDied","Data":"e3fd50398234caff08de646f26e7ce9c04b1e93c756d0cbc247e37a3d0f4a762"}
Nov 02 07:49:00.581999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.581970    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:49:00.603523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.603496    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:49:00.696123 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.696083    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\" (UniqueName: \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\") pod \"80e054c4-97a5-447a-a16f-15a004745833\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") "
Nov 02 07:49:00.696329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.696158    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\" (UniqueName: \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\") pod \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") "
Nov 02 07:49:00.696329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.696197    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\") pod \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\" (UID: \"ba8fd132-3171-4e4e-b1ed-5ff515e83621\") "
Nov 02 07:49:00.696471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.696384    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\") pod \"80e054c4-97a5-447a-a16f-15a004745833\" (UID: \"80e054c4-97a5-447a-a16f-15a004745833\") "
Nov 02 07:49:00.696745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.696709    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume" pod "ba8fd132-3171-4e4e-b1ed-5ff515e83621" (UID: "ba8fd132-3171-4e4e-b1ed-5ff515e83621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:00.697076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.697042    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume" pod "80e054c4-97a5-447a-a16f-15a004745833" (UID: "80e054c4-97a5-447a-a16f-15a004745833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:00.697291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.697265    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e054c4-97a5-447a-a16f-15a004745833-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:00.697412 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.697298    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8fd132-3171-4e4e-b1ed-5ff515e83621-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:00.698533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.698511    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd" pod "80e054c4-97a5-447a-a16f-15a004745833" (UID: "80e054c4-97a5-447a-a16f-15a004745833"). InnerVolumeSpecName "kube-api-access-2rzrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:00.698822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.698804    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf" pod "ba8fd132-3171-4e4e-b1ed-5ff515e83621" (UID: "ba8fd132-3171-4e4e-b1ed-5ff515e83621"). InnerVolumeSpecName "kube-api-access-svlbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:00.798301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.798270    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzrd\" (UniqueName: \"kubernetes.io/projected/80e054c4-97a5-447a-a16f-15a004745833-kube-api-access-2rzrd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:00.798301 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:00.798299    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svlbf\" (UniqueName: \"kubernetes.io/projected/ba8fd132-3171-4e4e-b1ed-5ff515e83621-kube-api-access-svlbf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:01.469916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.469881    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"
Nov 02 07:49:01.470416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.469915    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk" event={"ID":"80e054c4-97a5-447a-a16f-15a004745833","Type":"ContainerDied","Data":"5b45b5ba0aa70033f9749c52586c1af231a87aac82597002fcc7900e287648e8"}
Nov 02 07:49:01.470416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.469982    4515 scope.go:122] "RemoveContainer" containerID="b5d2ea929df80cc6c5b9bc351992885683535c7ce2cd93fc7c6c97d620485ac1"
Nov 02 07:49:01.471677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.471661    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"
Nov 02 07:49:01.471784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.471701    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb" event={"ID":"ba8fd132-3171-4e4e-b1ed-5ff515e83621","Type":"ContainerDied","Data":"5b644e9ccfeb3e72d38531dd2c4a68c36a19710686b23030a4c2a66cf4a48b8b"}
Nov 02 07:49:01.480026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.480001    4515 scope.go:122] "RemoveContainer" containerID="e3fd50398234caff08de646f26e7ce9c04b1e93c756d0cbc247e37a3d0f4a762"
Nov 02 07:49:01.546715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.546670    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:49:01.554375 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.554350    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2023a-654d66dc7f-68dlk"]
Nov 02 07:49:01.589200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.589161    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:49:01.593270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:01.593244    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2023a-84bcd7967c-hsvxb"]
Nov 02 07:49:03.398852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:03.398809    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e054c4-97a5-447a-a16f-15a004745833" path="/var/lib/kubelet/pods/80e054c4-97a5-447a-a16f-15a004745833/volumes"
Nov 02 07:49:03.399648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:03.399618    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" path="/var/lib/kubelet/pods/ba8fd132-3171-4e4e-b1ed-5ff515e83621/volumes"
Nov 02 07:49:03.402939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:03.402916    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:49:07.171498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.171460    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:49:07.184415 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.184379    4515 cpu_manager.go:414] "containerMap: removing container" podUID="80e054c4-97a5-447a-a16f-15a004745833" containerName="payload-pre-processing"
Nov 02 07:49:07.184612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.184597    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e054c4-97a5-447a-a16f-15a004745833" containerName="payload-pre-processing"
Nov 02 07:49:07.184739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.184725    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" containerName="payload-processing"
Nov 02 07:49:07.184819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.184809    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" containerName="payload-processing"
Nov 02 07:49:07.185138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.185123    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e054c4-97a5-447a-a16f-15a004745833" containerName="payload-pre-processing"
Nov 02 07:49:07.185241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.185231    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8fd132-3171-4e4e-b1ed-5ff515e83621" containerName="payload-processing"
Nov 02 07:49:07.185358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.185345    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1468920-9ced-4001-a0e4-4721f031c853" containerName="istio-proxy"
Nov 02 07:49:07.191570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.191541    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.207794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.207742    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-16e21e\""
Nov 02 07:49:07.213879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.213843    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:49:07.311888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.309680    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:49:07.319166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.319123    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfb7\" (UniqueName: \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.319473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.319432    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.321520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.320767    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.337057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.336912    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-16e21e-dockercfg-25d4p\""
Nov 02 07:49:07.339240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.339178    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:49:07.421062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.421020    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.421257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.421124    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhk74\" (UniqueName: \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.421468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.421441    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.421641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.421592    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfb7\" (UniqueName: \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.421913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.421890    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.446215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.446170    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfb7\" (UniqueName: \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\") pod \"payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.512976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.512929    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:07.523671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.523644    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhk74\" (UniqueName: \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.523833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.523782    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.524515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.524493    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.548413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.548380    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhk74\" (UniqueName: \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\") pod \"payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.655349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.652919    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:07.834821 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.834788    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:49:07.836745 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:49:07.836720    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd46948_365f_4741_b531_5d9b598daeff.slice/crio-c156eb5bc6d399e76b8dddb8111003a1d80d924c605966c87ea2ace5f332cab0 WatchSource:0}: Error finding container c156eb5bc6d399e76b8dddb8111003a1d80d924c605966c87ea2ace5f332cab0: Status 404 returned error can't find the container with id c156eb5bc6d399e76b8dddb8111003a1d80d924c605966c87ea2ace5f332cab0
Nov 02 07:49:07.976775 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:49:07.976733    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903e89cc_a05d_41cf_941e_539b23a06e4a.slice/crio-e02a534601298721c0d0d6586b05a72df4599f0e5c10cdcdcb69d7ca62785908 WatchSource:0}: Error finding container e02a534601298721c0d0d6586b05a72df4599f0e5c10cdcdcb69d7ca62785908: Status 404 returned error can't find the container with id e02a534601298721c0d0d6586b05a72df4599f0e5c10cdcdcb69d7ca62785908
Nov 02 07:49:07.976909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:07.976774    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:49:08.343366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.343331    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:49:08.343967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.343571    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy" containerID="cri-o://3439eacb87b4f5f0685986c914e1ea5b986faea4d2cfc2c7025625629e46fc88" gracePeriod=30
Nov 02 07:49:08.567545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.567208    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" event={"ID":"903e89cc-a05d-41cf-941e-539b23a06e4a","Type":"ContainerStarted","Data":"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"}
Nov 02 07:49:08.567545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.567270    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" event={"ID":"903e89cc-a05d-41cf-941e-539b23a06e4a","Type":"ContainerStarted","Data":"e02a534601298721c0d0d6586b05a72df4599f0e5c10cdcdcb69d7ca62785908"}
Nov 02 07:49:08.567545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.567467    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:08.569006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.568968    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" event={"ID":"ddd46948-365f-4741-b531-5d9b598daeff","Type":"ContainerStarted","Data":"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"}
Nov 02 07:49:08.569148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.569016    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" event={"ID":"ddd46948-365f-4741-b531-5d9b598daeff","Type":"ContainerStarted","Data":"c156eb5bc6d399e76b8dddb8111003a1d80d924c605966c87ea2ace5f332cab0"}
Nov 02 07:49:08.569148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.569102    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:08.743025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.742913    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" podStartSLOduration=1.742888739 podStartE2EDuration="1.742888739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:08.727883632 +0000 UTC m=+1493.490320558" watchObservedRunningTime="2026-11-02 07:49:08.742888739 +0000 UTC m=+1493.505325648"
Nov 02 07:49:08.766332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:08.757164    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" podStartSLOduration=1.757141031 podStartE2EDuration="1.757141031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:08.682380658 +0000 UTC m=+1493.444817565" watchObservedRunningTime="2026-11-02 07:49:08.757141031 +0000 UTC m=+1493.519577938"
Nov 02 07:49:09.734478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.734444    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:49:09.738950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.738927    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.770083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.770054    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eb64ad-serving-cert\""
Nov 02 07:49:09.776142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.776113    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:49:09.833164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.833120    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn87r\" (UniqueName: \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.833414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.833266    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.833414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.833383    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.833519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.833498    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.951587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.950907    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.951587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.951068    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn87r\" (UniqueName: \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.951587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.951157    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.951587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.951293    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.963357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.957529    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:09.996047 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:09.995940    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn87r\" (UniqueName: \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:10.019325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.017737    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:10.019325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.018160    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:10.055464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.055433    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:10.155120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.153405    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": read tcp 10.133.0.2:52078->10.133.0.145:15021: read: connection reset by peer" start-of-body=
Nov 02 07:49:10.155120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.153474    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": read tcp 10.133.0.2:52078->10.133.0.145:15021: read: connection reset by peer"
Nov 02 07:49:10.364255 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:49:10.364222    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79442b48_e4b1_4704_a794_1c4077b9f5b9.slice/crio-a235b2d43b8f13ea6c204f989ebd1759f0250050d7474c68a2ecfe6816a673cd WatchSource:0}: Error finding container a235b2d43b8f13ea6c204f989ebd1759f0250050d7474c68a2ecfe6816a673cd: Status 404 returned error can't find the container with id a235b2d43b8f13ea6c204f989ebd1759f0250050d7474c68a2ecfe6816a673cd
Nov 02 07:49:10.366015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.365988    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:49:10.585889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.585857    4515 generic.go:356] "Generic (PLEG): container finished" podID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerID="db12f1c23ae5f698e6142b7d53b8b32bf9f9b786308144787fc7faddbdc76142" exitCode=0
Nov 02 07:49:10.586067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.585929    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerDied","Data":"db12f1c23ae5f698e6142b7d53b8b32bf9f9b786308144787fc7faddbdc76142"}
Nov 02 07:49:10.586439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.586421    4515 scope.go:122] "RemoveContainer" containerID="db12f1c23ae5f698e6142b7d53b8b32bf9f9b786308144787fc7faddbdc76142"
Nov 02 07:49:10.587471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.587443    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" event={"ID":"79442b48-e4b1-4704-a794-1c4077b9f5b9","Type":"ContainerStarted","Data":"a235b2d43b8f13ea6c204f989ebd1759f0250050d7474c68a2ecfe6816a673cd"}
Nov 02 07:49:10.588434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.588398    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:10.588547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.588479    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:10.588547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:10.588517    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:11.606795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:11.606756    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerStarted","Data":"786cd89973a8eeb6e8d800ce75b7d44210fc52474b923c451c6ca3d4993aca06"}
Nov 02 07:49:11.609268 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:11.609240    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" event={"ID":"79442b48-e4b1-4704-a794-1c4077b9f5b9","Type":"ContainerStarted","Data":"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"}
Nov 02 07:49:11.609863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:11.609846    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:11.711078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:11.711018    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" podStartSLOduration=2.353497142 podStartE2EDuration="2.711004915s" totalImagesPullingTime="357.507773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:11.709207814 +0000 UTC m=+1496.471644745" watchObservedRunningTime="2026-11-02 07:49:11.711004915 +0000 UTC m=+1496.473441820"
Nov 02 07:49:12.404699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:12.404667    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:49:12.404900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:12.404874    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:49:12.405004 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:12.404932    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 02 07:49:13.405139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.405107    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:49:13.405659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.405173    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 02 07:49:13.651764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.651729    4515 generic.go:356] "Generic (PLEG): container finished" podID="0eaad616-829f-48e7-b900-46b36aa1922d" containerID="3439eacb87b4f5f0685986c914e1ea5b986faea4d2cfc2c7025625629e46fc88" exitCode=0
Nov 02 07:49:13.651882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.651771    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" event={"ID":"0eaad616-829f-48e7-b900-46b36aa1922d","Type":"ContainerDied","Data":"3439eacb87b4f5f0685986c914e1ea5b986faea4d2cfc2c7025625629e46fc88"}
Nov 02 07:49:13.681704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.681675    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:49:13.774835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.774797    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.774835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.774840    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.774922    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.774963    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775015    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775122    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775187    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775338    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775442 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775395    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\") pod \"0eaad616-829f-48e7-b900-46b36aa1922d\" (UID: \"0eaad616-829f-48e7-b900-46b36aa1922d\") "
Nov 02 07:49:13.775590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775562    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:13.775739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.775720    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:13.776606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.776199    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.776606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.776224    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.776606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.776382    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:13.776816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.776731    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:13.776978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.776946    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:13.777623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.777597    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:13.778162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.778136    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "kube-api-access-xvr9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:13.779223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.779201    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:13.779671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.779653    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo" pod "0eaad616-829f-48e7-b900-46b36aa1922d" (UID: "0eaad616-829f-48e7-b900-46b36aa1922d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882223    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eaad616-829f-48e7-b900-46b36aa1922d-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882261    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882275    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882288    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882301    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvr9r\" (UniqueName: \"kubernetes.io/projected/0eaad616-829f-48e7-b900-46b36aa1922d-kube-api-access-xvr9r\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882352    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eaad616-829f-48e7-b900-46b36aa1922d-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:13.882826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:13.882367    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eaad616-829f-48e7-b900-46b36aa1922d-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:14.405206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.405171    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:49:14.405648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.405238    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 02 07:49:14.661369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.661216    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf" event={"ID":"0eaad616-829f-48e7-b900-46b36aa1922d","Type":"ContainerDied","Data":"01d3b8f537e5d9e64a1938f75a5f703f13577ff4e18d080c03fa0e76501edf0e"}
Nov 02 07:49:14.661369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.661353    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"
Nov 02 07:49:14.661619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.661439    4515 scope.go:122] "RemoveContainer" containerID="3439eacb87b4f5f0685986c914e1ea5b986faea4d2cfc2c7025625629e46fc88"
Nov 02 07:49:14.715719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.715688    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:49:14.743511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:14.743466    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ncrlf"]
Nov 02 07:49:15.071696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.071660    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:49:15.072358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.072343    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy"
Nov 02 07:49:15.072404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.072360    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy"
Nov 02 07:49:15.072497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.072488    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" containerName="istio-proxy"
Nov 02 07:49:15.077529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.077510    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.093734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.093707    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-11b2e0\""
Nov 02 07:49:15.098552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.098527    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:49:15.243332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.238443    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q7jj\" (UniqueName: \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.271331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.258660    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.369381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.364603    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.369381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.364765    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q7jj\" (UniqueName: \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.369381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.366078    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.410335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.407482    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:49:15.410335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.407545    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 02 07:49:15.414382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.412208    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eaad616-829f-48e7-b900-46b36aa1922d" path="/var/lib/kubelet/pods/0eaad616-829f-48e7-b900-46b36aa1922d/volumes"
Nov 02 07:49:15.417070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.415041    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q7jj\" (UniqueName: \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\") pod \"payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:15.691649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:15.691563    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:16.044129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.044093    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:49:16.046863 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:49:16.046831    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2f9d27_4b21_42fe_9219_e996b1d1097f.slice/crio-17c5a2d75abe98e2e500a903db0a3908f354eb65117573a767ac80c752f2fe0a WatchSource:0}: Error finding container 17c5a2d75abe98e2e500a903db0a3908f354eb65117573a767ac80c752f2fe0a: Status 404 returned error can't find the container with id 17c5a2d75abe98e2e500a903db0a3908f354eb65117573a767ac80c752f2fe0a
Nov 02 07:49:16.404886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.404791    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:49:16.404886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.404853    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 02 07:49:16.709390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.702293    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" event={"ID":"fa2f9d27-4b21-42fe-9219-e996b1d1097f","Type":"ContainerStarted","Data":"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"}
Nov 02 07:49:16.709390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.702379    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" event={"ID":"fa2f9d27-4b21-42fe-9219-e996b1d1097f","Type":"ContainerStarted","Data":"17c5a2d75abe98e2e500a903db0a3908f354eb65117573a767ac80c752f2fe0a"}
Nov 02 07:49:16.709390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:16.702932    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:17.411379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:17.411346    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:49:17.411706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:17.411678    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:49:17.412733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:17.412705    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:49:17.482656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:17.482600    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" podStartSLOduration=2.482588012 podStartE2EDuration="2.482588012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:16.761862459 +0000 UTC m=+1501.524299365" watchObservedRunningTime="2026-11-02 07:49:17.482588012 +0000 UTC m=+1502.245024917"
Nov 02 07:49:17.725181 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:17.725094    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:49:18.657625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:18.657598    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:49:18.765407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:18.765373    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:49:18.766168 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:18.766130    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" podUID="826818f4-110a-451e-81bf-bccb21729021" containerName="maas-api" containerID="cri-o://4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5" gracePeriod=30
Nov 02 07:49:19.152935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.152902    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:49:19.179219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.179181    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\") pod \"826818f4-110a-451e-81bf-bccb21729021\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") "
Nov 02 07:49:19.179424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.179405    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\" (UniqueName: \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\") pod \"826818f4-110a-451e-81bf-bccb21729021\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") "
Nov 02 07:49:19.179548 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.179512    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") pod \"826818f4-110a-451e-81bf-bccb21729021\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") "
Nov 02 07:49:19.179867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.179832    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca" pod "826818f4-110a-451e-81bf-bccb21729021" (UID: "826818f4-110a-451e-81bf-bccb21729021"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:19.179955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.179927    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\") pod \"826818f4-110a-451e-81bf-bccb21729021\" (UID: \"826818f4-110a-451e-81bf-bccb21729021\") "
Nov 02 07:49:19.180597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.180574    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826818f4-110a-451e-81bf-bccb21729021-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:19.183902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.183860    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j" pod "826818f4-110a-451e-81bf-bccb21729021" (UID: "826818f4-110a-451e-81bf-bccb21729021"). InnerVolumeSpecName "kube-api-access-mhv6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:19.184374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.184349    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs" pod "826818f4-110a-451e-81bf-bccb21729021" (UID: "826818f4-110a-451e-81bf-bccb21729021"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:19.185120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.185085    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls" pod "826818f4-110a-451e-81bf-bccb21729021" (UID: "826818f4-110a-451e-81bf-bccb21729021"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:19.281746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.281709    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhv6j\" (UniqueName: \"kubernetes.io/projected/826818f4-110a-451e-81bf-bccb21729021-kube-api-access-mhv6j\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:19.281746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.281740    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:19.281746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.281749    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826818f4-110a-451e-81bf-bccb21729021-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:19.579506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.579471    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:49:19.579688 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.579564    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:49:19.733691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.733657    4515 generic.go:356] "Generic (PLEG): container finished" podID="826818f4-110a-451e-81bf-bccb21729021" containerID="4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5" exitCode=0
Nov 02 07:49:19.733901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.733715    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"
Nov 02 07:49:19.733901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.733742    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" event={"ID":"826818f4-110a-451e-81bf-bccb21729021","Type":"ContainerDied","Data":"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"}
Nov 02 07:49:19.733901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.733796    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr" event={"ID":"826818f4-110a-451e-81bf-bccb21729021","Type":"ContainerDied","Data":"0e6079906217313ba8349e712fb2202b1b46f1bc37d529e546c34b9d3ad60239"}
Nov 02 07:49:19.733901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.733820    4515 scope.go:122] "RemoveContainer" containerID="4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"
Nov 02 07:49:19.744405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.744383    4515 scope.go:122] "RemoveContainer" containerID="4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"
Nov 02 07:49:19.744722 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:19.744700    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5\": container with ID starting with 4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5 not found: ID does not exist" containerID="4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"
Nov 02 07:49:19.744775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.744730    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5"} err="failed to get container status \"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5\": rpc error: code = NotFound desc = could not find container \"4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5\": container with ID starting with 4f510e18e38d6513b053d86525283f4da0165c6f5200c9f84aa454f01d4229c5 not found: ID does not exist"
Nov 02 07:49:19.841270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.841172    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:49:19.854566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:19.854156    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2023a-7cdd645574-gsrvr"]
Nov 02 07:49:21.078577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.078538    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:49:21.079221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.079199    4515 cpu_manager.go:414] "containerMap: removing container" podUID="826818f4-110a-451e-81bf-bccb21729021" containerName="maas-api"
Nov 02 07:49:21.079221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.079217    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826818f4-110a-451e-81bf-bccb21729021" containerName="maas-api"
Nov 02 07:49:21.079417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.079395    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="826818f4-110a-451e-81bf-bccb21729021" containerName="maas-api"
Nov 02 07:49:21.084977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.084836    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.106773 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.104809    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-677726-openshift-default-dockercfg-pwnw5\""
Nov 02 07:49:21.125944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.125913    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:49:21.134275 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134239    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134295    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134364    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134430    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134518    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpkm\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134740    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.134841 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.134811    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240162    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240256    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240351    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240396    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240431    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240463    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240547    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.240661    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpkm\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.241817    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.242095    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.242338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.242582    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.244381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.243066    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.277166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.260745    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.294342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.288634    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpkm\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.317179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.317143    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.323382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.319108    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\") pod \"e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") " pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.412195 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.411792    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826818f4-110a-451e-81bf-bccb21729021" path="/var/lib/kubelet/pods/826818f4-110a-451e-81bf-bccb21729021/volumes"
Nov 02 07:49:21.438333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.429614    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:21.778359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.778183    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:49:21.780723 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.780689    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:21.780817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.780754    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:21.780817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:21.780789    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:22.756191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:22.756153    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerStarted","Data":"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"}
Nov 02 07:49:22.756640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:22.756252    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerStarted","Data":"ef54692b30411d3c2e12a49424ae9f42cd25b083eb58f43b3e8816be385db065"}
Nov 02 07:49:22.817462 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:22.817392    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podStartSLOduration=1.817373855 podStartE2EDuration="1.817373855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:22.814822255 +0000 UTC m=+1507.577259160" watchObservedRunningTime="2026-11-02 07:49:22.817373855 +0000 UTC m=+1507.579810761"
Nov 02 07:49:23.438808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:23.438778    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:23.444304 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:23.444278    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:23.763500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:23.763471    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:23.764575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:23.764549    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:49:27.714288 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:27.714259    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:49:28.638952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:28.638917    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:49:28.639197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:28.639172    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" containerID="cri-o://23abdf782b9e0320103f5ecdb73cf5149a8aab43d9f131a4caaae18b657dbcb6" gracePeriod=30
Nov 02 07:49:30.234000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:30.233956    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:49:33.391578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:33.391542    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:49:33.392010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:33.391605    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:49:33.861769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:33.861737    4515 generic.go:356] "Generic (PLEG): container finished" podID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerID="23abdf782b9e0320103f5ecdb73cf5149a8aab43d9f131a4caaae18b657dbcb6" exitCode=0
Nov 02 07:49:33.861945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:33.861808    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerDied","Data":"23abdf782b9e0320103f5ecdb73cf5149a8aab43d9f131a4caaae18b657dbcb6"}
Nov 02 07:49:33.861945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:33.861876    4515 scope.go:122] "RemoveContainer" containerID="8007e6257ffbe6a24795c5b49de232c518c443099956da4185af4d66d2d7ad7c"
Nov 02 07:49:34.146083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.146042    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:49:34.180712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.180682    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:49:34.354539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.354514    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:49:34.432103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.419596    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.432103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.420112    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.432103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.420165    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.432103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.420231    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.432103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.420276    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.432934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.432407    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:34.434045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.433718    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:34.460131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.459424    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.460131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.459485    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.460131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.459540    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.460131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.459745    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\") pod \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\" (UID: \"264df5d7-7b99-4d40-9b6c-a3a4249a742c\") "
Nov 02 07:49:34.460131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.459912    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460149    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460420    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460694    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460720    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460735    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460750    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.460805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.460764    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.470448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.467426    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:34.470448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.468451    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:49:34.470448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.468588    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "kube-api-access-22rzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:34.478262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.478231    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token" pod "264df5d7-7b99-4d40-9b6c-a3a4249a742c" (UID: "264df5d7-7b99-4d40-9b6c-a3a4249a742c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:34.561752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.561724    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.561752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.561749    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rzv\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-kube-api-access-22rzv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.561752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.561762    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.561986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.561771    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264df5d7-7b99-4d40-9b6c-a3a4249a742c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:34.889331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.888636    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"
Nov 02 07:49:34.889807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.888642    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28" event={"ID":"264df5d7-7b99-4d40-9b6c-a3a4249a742c","Type":"ContainerDied","Data":"a947db9895a091e8d2288a2788f2faeb29bb042a70fdf9888d88c4a647c9b341"}
Nov 02 07:49:34.890162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.889993    4515 scope.go:122] "RemoveContainer" containerID="23abdf782b9e0320103f5ecdb73cf5149a8aab43d9f131a4caaae18b657dbcb6"
Nov 02 07:49:34.959895 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.959853    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:49:34.977303 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:34.974757    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2023a-openshift-default-d7cc9c877-v2w28"]
Nov 02 07:49:35.374260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:35.374226    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" path="/var/lib/kubelet/pods/264df5d7-7b99-4d40-9b6c-a3a4249a742c/volumes"
Nov 02 07:49:37.489202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.489160    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:49:37.490008 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.489986    4515 cpu_manager.go:414] "containerMap: removing container" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:37.490115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.490010    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:37.490115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.490072    4515 cpu_manager.go:414] "containerMap: removing container" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:37.490115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.490080    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:37.490283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.490268    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:37.493978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.493956    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.511642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.511606    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.511817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.511791    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.511961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.511941    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxsq\" (UniqueName: \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.512074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.512056    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.525057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.525026    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-11b2e0-serving-cert\""
Nov 02 07:49:37.530571 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.530541    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:49:37.613150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.613118    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.613359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.613197    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.613359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.613239    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxsq\" (UniqueName: \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.613359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.613283    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.613545 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:37.613477    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-serving-cert: secret "maas-api-e2e-praxis-b-11b2e0-serving-cert" not found
Nov 02 07:49:37.613545 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:37.613544    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls podName:c3f3f36e-4408-4a22-a511-1dfcf9a50b9f nodeName:}" failed. No retries permitted until 2026-11-02 07:49:38.113529138 +0000 UTC m=+1522.875966022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls") pod "maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" (UID: "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f") : secret "maas-api-e2e-praxis-b-11b2e0-serving-cert" not found
Nov 02 07:49:37.613998 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.613970    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.615939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.615916    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:37.637796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:37.637767    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxsq\" (UniqueName: \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:38.120831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:38.120791    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:38.123279 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:38.123258    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:38.407671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:38.407580    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:38.764807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:38.762899    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:49:38.923191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:38.923156    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" event={"ID":"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f","Type":"ContainerStarted","Data":"bcb20c09102ce8590bae3e2006b8d09cb7ca590ec13075902ec4e5d8645fb59e"}
Nov 02 07:49:39.147209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.147167    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" podUID="355898c2-3263-4bf9-a9a2-2c1f46997c28" containerName="payload-processing" containerID="cri-o://6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e" gracePeriod=25
Nov 02 07:49:39.181445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.181409    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" podUID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" containerName="payload-pre-processing" containerID="cri-o://5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61" gracePeriod=25
Nov 02 07:49:39.326119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.325992    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:49:39.326632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.326600    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" podUID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" containerName="maas-api" containerID="cri-o://c3e10f077ceb3ca1ea5880f4cb551ada92009eb68e02a5651ff978e8c88d0254" gracePeriod=30
Nov 02 07:49:39.687675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.687598    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:49:39.694251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.694223    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:49:39.788582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.788551    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\") pod \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") "
Nov 02 07:49:39.789544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.789524    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\") pod \"355898c2-3263-4bf9-a9a2-2c1f46997c28\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") "
Nov 02 07:49:39.790351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.790334    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\" (UniqueName: \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\") pod \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\" (UID: \"3ba9cb59-5c5e-4e35-bccf-0339c8505179\") "
Nov 02 07:49:39.790792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.790776    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\" (UniqueName: \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\") pod \"355898c2-3263-4bf9-a9a2-2c1f46997c28\" (UID: \"355898c2-3263-4bf9-a9a2-2c1f46997c28\") "
Nov 02 07:49:39.791756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.789442    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume" pod "3ba9cb59-5c5e-4e35-bccf-0339c8505179" (UID: "3ba9cb59-5c5e-4e35-bccf-0339c8505179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:39.791961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.790085    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume" pod "355898c2-3263-4bf9-a9a2-2c1f46997c28" (UID: "355898c2-3263-4bf9-a9a2-2c1f46997c28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:39.793553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.793518    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp" pod "355898c2-3263-4bf9-a9a2-2c1f46997c28" (UID: "355898c2-3263-4bf9-a9a2-2c1f46997c28"). InnerVolumeSpecName "kube-api-access-2bwdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:39.795493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.795469    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7" pod "3ba9cb59-5c5e-4e35-bccf-0339c8505179" (UID: "3ba9cb59-5c5e-4e35-bccf-0339c8505179"). InnerVolumeSpecName "kube-api-access-l9fc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:39.892755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.892717    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba9cb59-5c5e-4e35-bccf-0339c8505179-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:39.892755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.892750    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355898c2-3263-4bf9-a9a2-2c1f46997c28-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:39.892755 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.892759    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fc7\" (UniqueName: \"kubernetes.io/projected/3ba9cb59-5c5e-4e35-bccf-0339c8505179-kube-api-access-l9fc7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:39.893112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.892770    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwdp\" (UniqueName: \"kubernetes.io/projected/355898c2-3263-4bf9-a9a2-2c1f46997c28-kube-api-access-2bwdp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:39.932180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.932142    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" event={"ID":"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f","Type":"ContainerStarted","Data":"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"}
Nov 02 07:49:39.932367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.932350    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:39.933708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.933677    4515 generic.go:356] "Generic (PLEG): container finished" podID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" containerID="5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61" exitCode=0
Nov 02 07:49:39.933833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.933721    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"
Nov 02 07:49:39.933833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.933757    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" event={"ID":"3ba9cb59-5c5e-4e35-bccf-0339c8505179","Type":"ContainerDied","Data":"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"}
Nov 02 07:49:39.933833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.933802    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm" event={"ID":"3ba9cb59-5c5e-4e35-bccf-0339c8505179","Type":"ContainerDied","Data":"4865a76d32e105a060e0ec4e6f2afa7b5c3b944d9ef0233238ec7d6efcc5dc1a"}
Nov 02 07:49:39.933833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.933825    4515 scope.go:122] "RemoveContainer" containerID="5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"
Nov 02 07:49:39.935368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.935345    4515 generic.go:356] "Generic (PLEG): container finished" podID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" containerID="c3e10f077ceb3ca1ea5880f4cb551ada92009eb68e02a5651ff978e8c88d0254" exitCode=0
Nov 02 07:49:39.935511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.935406    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" event={"ID":"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e","Type":"ContainerDied","Data":"c3e10f077ceb3ca1ea5880f4cb551ada92009eb68e02a5651ff978e8c88d0254"}
Nov 02 07:49:39.935511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.935443    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb" event={"ID":"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e","Type":"ContainerDied","Data":"88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10"}
Nov 02 07:49:39.935511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.935464    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88a156b3bd5f65a362097162083a5be795674cb9e6c5494d53f44d6ea8f6af10"
Nov 02 07:49:39.937027 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.936996    4515 generic.go:356] "Generic (PLEG): container finished" podID="355898c2-3263-4bf9-a9a2-2c1f46997c28" containerID="6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e" exitCode=0
Nov 02 07:49:39.937152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.937037    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" event={"ID":"355898c2-3263-4bf9-a9a2-2c1f46997c28","Type":"ContainerDied","Data":"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"}
Nov 02 07:49:39.937152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.937076    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7" event={"ID":"355898c2-3263-4bf9-a9a2-2c1f46997c28","Type":"ContainerDied","Data":"4e879cc575ad76362a02b840b0aeee1e2e4e9b1f0e8a59065f7a7e47b1c973b6"}
Nov 02 07:49:39.937265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.937178    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"
Nov 02 07:49:39.942154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.942104    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:49:39.955289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:39.950584    4515 scope.go:122] "RemoveContainer" containerID="5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"
Nov 02 07:49:40.029714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.028871    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" podStartSLOduration=2.680257638 podStartE2EDuration="3.02885243s" totalImagesPullingTime="348.594792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:40.026328121 +0000 UTC m=+1524.788765038" watchObservedRunningTime="2026-11-02 07:49:40.02885243 +0000 UTC m=+1524.791289337"
Nov 02 07:49:40.029714 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:40.029482    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61\": container with ID starting with 5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61 not found: ID does not exist" containerID="5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"
Nov 02 07:49:40.029714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.029531    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61"} err="failed to get container status \"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61\": rpc error: code = NotFound desc = could not find container \"5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61\": container with ID starting with 5c40c53351f17c547e7e2143fc960e158d4a4031d155c7cf28b6191df4018d61 not found: ID does not exist"
Nov 02 07:49:40.029714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.029558    4515 scope.go:122] "RemoveContainer" containerID="6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"
Nov 02 07:49:40.056432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.055604    4515 scope.go:122] "RemoveContainer" containerID="6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"
Nov 02 07:49:40.062845 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:40.062810    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e\": container with ID starting with 6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e not found: ID does not exist" containerID="6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"
Nov 02 07:49:40.063096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.063059    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e"} err="failed to get container status \"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e\": rpc error: code = NotFound desc = could not find container \"6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e\": container with ID starting with 6cb0818a18dd73f1b48d63ed8404574436188321cb361d038e1cdc1ccb2efc4e not found: ID does not exist"
Nov 02 07:49:40.103473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.096148    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\") pod \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") "
Nov 02 07:49:40.103473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.096251    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\" (UniqueName: \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\") pod \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") "
Nov 02 07:49:40.103473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.096287    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\") pod \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") "
Nov 02 07:49:40.103473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.096364    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\") pod \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\" (UID: \"54b5c0f2-130d-4acd-a1ee-7f17438d2a5e\") "
Nov 02 07:49:40.114561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.114519    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca" pod "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" (UID: "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:40.116748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.116712    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs" pod "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" (UID: "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:40.133706 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.132844    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:49:40.139814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.139754    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4" pod "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" (UID: "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e"). InnerVolumeSpecName "kube-api-access-7jnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:40.140345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.140299    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls" pod "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" (UID: "54b5c0f2-130d-4acd-a1ee-7f17438d2a5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:40.154302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.154269    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d60b29-758d9497c7-mqhd7"]
Nov 02 07:49:40.163699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.163665    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:49:40.167395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.167370    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d60b29-6b5cdf4959-fdvcm"]
Nov 02 07:49:40.223494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.202879    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:40.223494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.202918    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:40.223870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.223776    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnw4\" (UniqueName: \"kubernetes.io/projected/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-kube-api-access-7jnw4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:40.223870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.223839    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:40.946262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:40.946234    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"
Nov 02 07:49:41.009610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:41.009575    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:49:41.022260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:41.022170    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d60b29-db67969db-p47hb"]
Nov 02 07:49:41.419402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:41.417690    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355898c2-3263-4bf9-a9a2-2c1f46997c28" path="/var/lib/kubelet/pods/355898c2-3263-4bf9-a9a2-2c1f46997c28/volumes"
Nov 02 07:49:41.419402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:41.418417    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" path="/var/lib/kubelet/pods/3ba9cb59-5c5e-4e35-bccf-0339c8505179/volumes"
Nov 02 07:49:41.419402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:41.419090    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" path="/var/lib/kubelet/pods/54b5c0f2-130d-4acd-a1ee-7f17438d2a5e/volumes"
Nov 02 07:49:45.951482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:45.951449    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:49:47.094411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:47.094375    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:49:47.094871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:47.094732    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" containerID="cri-o://25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482" gracePeriod=30
Nov 02 07:49:47.723598 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:47.723555    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:49:48.401132 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:48.401096    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:49:48.401595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:48.401150    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:49:50.351532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.351497    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:49:50.352161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352145    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" containerName="payload-pre-processing"
Nov 02 07:49:50.352214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352163    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" containerName="payload-pre-processing"
Nov 02 07:49:50.352214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352176    4515 cpu_manager.go:414] "containerMap: removing container" podUID="355898c2-3263-4bf9-a9a2-2c1f46997c28" containerName="payload-processing"
Nov 02 07:49:50.352214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352181    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355898c2-3263-4bf9-a9a2-2c1f46997c28" containerName="payload-processing"
Nov 02 07:49:50.352214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352211    4515 cpu_manager.go:414] "containerMap: removing container" podUID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" containerName="maas-api"
Nov 02 07:49:50.352394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352216    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" containerName="maas-api"
Nov 02 07:49:50.352394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352302    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba9cb59-5c5e-4e35-bccf-0339c8505179" containerName="payload-pre-processing"
Nov 02 07:49:50.352394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352325    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="355898c2-3263-4bf9-a9a2-2c1f46997c28" containerName="payload-processing"
Nov 02 07:49:50.352394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352336    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b5c0f2-130d-4acd-a1ee-7f17438d2a5e" containerName="maas-api"
Nov 02 07:49:50.352394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.352347    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="264df5d7-7b99-4d40-9b6c-a3a4249a742c" containerName="istio-proxy"
Nov 02 07:49:50.357289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.357269    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.375851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.375820    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:49:50.388917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.384889    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-677726-serving-cert\""
Nov 02 07:49:50.430633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.430599    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.430808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.430661    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl2b\" (UniqueName: \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.430808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.430753    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.430902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.430862    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.531770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.531738    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl2b\" (UniqueName: \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.531940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.531793    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.531940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.531895    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.532097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.532079    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.532215 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:50.532197    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-serving-cert: secret "maas-api-e2e-ipp-b-677726-serving-cert" not found
Nov 02 07:49:50.532289 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:50.532269    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls podName:18324e8c-3844-429c-ae27-605c435ea14b nodeName:}" failed. No retries permitted until 2026-11-02 07:49:51.032248684 +0000 UTC m=+1535.794685570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls") pod "maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" (UID: "18324e8c-3844-429c-ae27-605c435ea14b") : secret "maas-api-e2e-ipp-b-677726-serving-cert" not found
Nov 02 07:49:50.532645 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.532623    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.551420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.546997    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:50.581339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:50.580086    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl2b\" (UniqueName: \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:51.044091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:51.044057    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:51.046543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:51.046514    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:51.272185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:51.272151    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:51.630601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:51.630446    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:49:52.051884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.051854    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" event={"ID":"18324e8c-3844-429c-ae27-605c435ea14b","Type":"ContainerStarted","Data":"8b436d67a00c878fbf281e17794f15b0d5187921a74739c046afce717f382445"}
Nov 02 07:49:52.418408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.418051    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:49:52.520040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520000    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520235 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520164    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520235 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520208    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520381    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520469    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520513    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520624    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520689    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520623    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:52.520757 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520724    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\") pod \"289c2eb7-77ac-4258-9a07-bb383140ac56\" (UID: \"289c2eb7-77ac-4258-9a07-bb383140ac56\") "
Nov 02 07:49:52.520956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.520934    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:52.521287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521261    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:52.521423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521392    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.521423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521417    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.521547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521431    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c2eb7-77ac-4258-9a07-bb383140ac56-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.521609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521540    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:52.521698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.521674    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:52.523253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.523225    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:52.523973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.523880    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:49:52.523973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.523913    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "kube-api-access-6t9cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:52.523973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.523953    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token" pod "289c2eb7-77ac-4258-9a07-bb383140ac56" (UID: "289c2eb7-77ac-4258-9a07-bb383140ac56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622738    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622769    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622779    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622789    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622797    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c2eb7-77ac-4258-9a07-bb383140ac56-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:52.622820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:52.622806    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9cb\" (UniqueName: \"kubernetes.io/projected/289c2eb7-77ac-4258-9a07-bb383140ac56-kube-api-access-6t9cb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:53.059984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.059946    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" event={"ID":"18324e8c-3844-429c-ae27-605c435ea14b","Type":"ContainerStarted","Data":"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"}
Nov 02 07:49:53.060452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.060017    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:53.061819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.061788    4515 generic.go:356] "Generic (PLEG): container finished" podID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerID="25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482" exitCode=0
Nov 02 07:49:53.061943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.061833    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"
Nov 02 07:49:53.061943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.061871    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerDied","Data":"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"}
Nov 02 07:49:53.061943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.061918    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77" event={"ID":"289c2eb7-77ac-4258-9a07-bb383140ac56","Type":"ContainerDied","Data":"aca20250ed4963dcc404a9dacc6727fb5fefc2a934dc5d3f1095d80fc3db1a0f"}
Nov 02 07:49:53.061943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.061941    4515 scope.go:122] "RemoveContainer" containerID="25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"
Nov 02 07:49:53.072764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.072748    4515 scope.go:122] "RemoveContainer" containerID="463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"
Nov 02 07:49:53.082584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.082564    4515 scope.go:122] "RemoveContainer" containerID="25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"
Nov 02 07:49:53.082851 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:53.082828    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482\": container with ID starting with 25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482 not found: ID does not exist" containerID="25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"
Nov 02 07:49:53.082906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.082858    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482"} err="failed to get container status \"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482\": rpc error: code = NotFound desc = could not find container \"25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482\": container with ID starting with 25d2c3148a4abd383495cece923fe6fde227c12ceb28f7bf101aa961efa78482 not found: ID does not exist"
Nov 02 07:49:53.082906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.082875    4515 scope.go:122] "RemoveContainer" containerID="463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"
Nov 02 07:49:53.083148 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:53.083128    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4\": container with ID starting with 463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4 not found: ID does not exist" containerID="463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"
Nov 02 07:49:53.083194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.083153    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4"} err="failed to get container status \"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4\": rpc error: code = NotFound desc = could not find container \"463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4\": container with ID starting with 463e35e373c3ce35dfb2e4bc594100e69f8348ab075adb023e5f9a9d9d455ce4 not found: ID does not exist"
Nov 02 07:49:53.104470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.104425    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" podStartSLOduration=2.708896769 podStartE2EDuration="3.104412828s" totalImagesPullingTime="395.516059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:53.102652719 +0000 UTC m=+1537.865089625" watchObservedRunningTime="2026-11-02 07:49:53.104412828 +0000 UTC m=+1537.866849731"
Nov 02 07:49:53.142090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.142063    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:49:53.154686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.154659    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d60b29-openshift-default-8f7bcb7f7-27t77"]
Nov 02 07:49:53.374669 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.374590    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" path="/var/lib/kubelet/pods/289c2eb7-77ac-4258-9a07-bb383140ac56/volumes"
Nov 02 07:49:53.546763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.546732    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:49:53.581545 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:53.581510    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:49:57.267650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:57.266170    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:49:58.547270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.547223    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" podUID="43299bc1-855b-40e1-a5d0-76f61a739f50" containerName="payload-processing" containerID="cri-o://3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28" gracePeriod=25
Nov 02 07:49:58.581809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.581770    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" podUID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" containerName="payload-pre-processing" containerID="cri-o://942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4" gracePeriod=25
Nov 02 07:49:58.886519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.886485    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:49:58.920331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.920288    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:49:58.960076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960044    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\" (UniqueName: \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\") pod \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") "
Nov 02 07:49:58.960296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960196    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\") pod \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\" (UID: \"160c29bf-a3c2-49f1-ad48-2b2579caf7cd\") "
Nov 02 07:49:58.960296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960221    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\") pod \"43299bc1-855b-40e1-a5d0-76f61a739f50\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") "
Nov 02 07:49:58.960296 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960262    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\" (UniqueName: \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\") pod \"43299bc1-855b-40e1-a5d0-76f61a739f50\" (UID: \"43299bc1-855b-40e1-a5d0-76f61a739f50\") "
Nov 02 07:49:58.961017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960750    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume" pod "160c29bf-a3c2-49f1-ad48-2b2579caf7cd" (UID: "160c29bf-a3c2-49f1-ad48-2b2579caf7cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:58.961017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.960898    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:58.961973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.961931    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume" pod "43299bc1-855b-40e1-a5d0-76f61a739f50" (UID: "43299bc1-855b-40e1-a5d0-76f61a739f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:58.963679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.963655    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb" pod "43299bc1-855b-40e1-a5d0-76f61a739f50" (UID: "43299bc1-855b-40e1-a5d0-76f61a739f50"). InnerVolumeSpecName "kube-api-access-qpmqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:58.964506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:58.964487    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58" pod "160c29bf-a3c2-49f1-ad48-2b2579caf7cd" (UID: "160c29bf-a3c2-49f1-ad48-2b2579caf7cd"). InnerVolumeSpecName "kube-api-access-6tx58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:59.062599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.062559    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43299bc1-855b-40e1-a5d0-76f61a739f50-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:59.062599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.062591    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpmqb\" (UniqueName: \"kubernetes.io/projected/43299bc1-855b-40e1-a5d0-76f61a739f50-kube-api-access-qpmqb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:59.062599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.062604    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tx58\" (UniqueName: \"kubernetes.io/projected/160c29bf-a3c2-49f1-ad48-2b2579caf7cd-kube-api-access-6tx58\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:49:59.078698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.078614    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:49:59.189559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.189521    4515 generic.go:356] "Generic (PLEG): container finished" podID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" containerID="942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4" exitCode=0
Nov 02 07:49:59.189752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.189573    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"
Nov 02 07:49:59.189752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.189727    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" event={"ID":"160c29bf-a3c2-49f1-ad48-2b2579caf7cd","Type":"ContainerDied","Data":"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"}
Nov 02 07:49:59.189886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.189788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf" event={"ID":"160c29bf-a3c2-49f1-ad48-2b2579caf7cd","Type":"ContainerDied","Data":"0db9dbf9a7da00885041a778d2ad044cad559863b0c43678d07338e8ec740398"}
Nov 02 07:49:59.189886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.189819    4515 scope.go:122] "RemoveContainer" containerID="942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"
Nov 02 07:49:59.193176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.193131    4515 generic.go:356] "Generic (PLEG): container finished" podID="43299bc1-855b-40e1-a5d0-76f61a739f50" containerID="3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28" exitCode=0
Nov 02 07:49:59.193436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.193415    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" event={"ID":"43299bc1-855b-40e1-a5d0-76f61a739f50","Type":"ContainerDied","Data":"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"}
Nov 02 07:49:59.193528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.193493    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x" event={"ID":"43299bc1-855b-40e1-a5d0-76f61a739f50","Type":"ContainerDied","Data":"69dc3afee3fa049cc47cac26f25438cd22263ef32637ff8c4fe1b0955194d242"}
Nov 02 07:49:59.193614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.193599    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"
Nov 02 07:49:59.205911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.205723    4515 scope.go:122] "RemoveContainer" containerID="942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"
Nov 02 07:49:59.207095 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:59.207066    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4\": container with ID starting with 942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4 not found: ID does not exist" containerID="942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"
Nov 02 07:49:59.207224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.207105    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4"} err="failed to get container status \"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4\": rpc error: code = NotFound desc = could not find container \"942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4\": container with ID starting with 942e75c6730dd809e21635fe4b6a0dd770520286933cd4f96b8ec2009b0055b4 not found: ID does not exist"
Nov 02 07:49:59.207224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.207130    4515 scope.go:122] "RemoveContainer" containerID="3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"
Nov 02 07:49:59.228478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.228452    4515 scope.go:122] "RemoveContainer" containerID="3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"
Nov 02 07:49:59.228981 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:49:59.228943    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28\": container with ID starting with 3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28 not found: ID does not exist" containerID="3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"
Nov 02 07:49:59.229096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.228992    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28"} err="failed to get container status \"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28\": rpc error: code = NotFound desc = could not find container \"3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28\": container with ID starting with 3c2650d6cc45b80f7d0f380417ba20e0f67a72f344c399ad1473c8ec1c3e0d28 not found: ID does not exist"
Nov 02 07:49:59.267334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.264553    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:49:59.268893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.268771    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b2f76f-67bcd7cb8c-4bksf"]
Nov 02 07:49:59.304806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.304773    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:49:59.309278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.309246    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b2f76f-576784f587-hk79x"]
Nov 02 07:49:59.385716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.385627    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" path="/var/lib/kubelet/pods/160c29bf-a3c2-49f1-ad48-2b2579caf7cd/volumes"
Nov 02 07:49:59.386165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:49:59.386143    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43299bc1-855b-40e1-a5d0-76f61a739f50" path="/var/lib/kubelet/pods/43299bc1-855b-40e1-a5d0-76f61a739f50/volumes"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.829018    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830198    4515 cpu_manager.go:414] "containerMap: removing container" podUID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" containerName="payload-pre-processing"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830221    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" containerName="payload-pre-processing"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830262    4515 cpu_manager.go:414] "containerMap: removing container" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830271    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830332    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43299bc1-855b-40e1-a5d0-76f61a739f50" containerName="payload-processing"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830340    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43299bc1-855b-40e1-a5d0-76f61a739f50" containerName="payload-processing"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830481    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="160c29bf-a3c2-49f1-ad48-2b2579caf7cd" containerName="payload-pre-processing"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830502    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830513    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.830741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.830527    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43299bc1-855b-40e1-a5d0-76f61a739f50" containerName="payload-processing"
Nov 02 07:50:03.846342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.838167    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:03.872472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.872424    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:03.883335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.878682    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb64ad\""
Nov 02 07:50:03.883335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.879460    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb64ad-dockercfg-g7sxs\""
Nov 02 07:50:03.938834 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.938796    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:03.939026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.938949    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:03.973598 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.973558    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:03.974202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.974188    4515 cpu_manager.go:414] "containerMap: removing container" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.974261 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.974204    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c2eb7-77ac-4258-9a07-bb383140ac56" containerName="istio-proxy"
Nov 02 07:50:03.977982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.977955    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:03.999079 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:03.999048    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:04.040238 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.040200    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.040481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.040254    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:04.040481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.040388    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5dj\" (UniqueName: \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.040481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.040458    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:04.041473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.041449    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:04.081466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.081385    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:04.141781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.141740    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.141992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.141798    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5dj\" (UniqueName: \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.142529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.142506    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.167344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.167279    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:04.186087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.186057    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5dj\" (UniqueName: \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\") pod \"payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.293819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.293787    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:04.473995 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:50:04.473965    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1742c837_da1e_4fed_b681_c0f2d7aeb385.slice/crio-b1a75501b6741b1d1ade4f7704e70944cd9f3fcf96a368b076e445ade0780ede WatchSource:0}: Error finding container b1a75501b6741b1d1ade4f7704e70944cd9f3fcf96a368b076e445ade0780ede: Status 404 returned error can't find the container with id b1a75501b6741b1d1ade4f7704e70944cd9f3fcf96a368b076e445ade0780ede
Nov 02 07:50:04.475091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.475069    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:04.606208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:04.606175    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:04.606735 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:50:04.606708    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075f8a70_0edc_42f3_9f1e_5f72b059f5bc.slice/crio-4630fa4a17bad115167f7f75a777dd9f2a5f78ee4665a87d388579db59d3932e WatchSource:0}: Error finding container 4630fa4a17bad115167f7f75a777dd9f2a5f78ee4665a87d388579db59d3932e: Status 404 returned error can't find the container with id 4630fa4a17bad115167f7f75a777dd9f2a5f78ee4665a87d388579db59d3932e
Nov 02 07:50:05.264781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.264737    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" event={"ID":"075f8a70-0edc-42f3-9f1e-5f72b059f5bc","Type":"ContainerStarted","Data":"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"}
Nov 02 07:50:05.265324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.264811    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" event={"ID":"075f8a70-0edc-42f3-9f1e-5f72b059f5bc","Type":"ContainerStarted","Data":"4630fa4a17bad115167f7f75a777dd9f2a5f78ee4665a87d388579db59d3932e"}
Nov 02 07:50:05.265324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.264860    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:05.266711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.266686    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" event={"ID":"1742c837-da1e-4fed-b681-c0f2d7aeb385","Type":"ContainerStarted","Data":"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"}
Nov 02 07:50:05.266820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.266727    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" event={"ID":"1742c837-da1e-4fed-b681-c0f2d7aeb385","Type":"ContainerStarted","Data":"b1a75501b6741b1d1ade4f7704e70944cd9f3fcf96a368b076e445ade0780ede"}
Nov 02 07:50:05.266859 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.266820    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:05.308497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.308441    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" podStartSLOduration=2.308424723 podStartE2EDuration="2.308424723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:05.306129607 +0000 UTC m=+1550.068566519" watchObservedRunningTime="2026-11-02 07:50:05.308424723 +0000 UTC m=+1550.070861644"
Nov 02 07:50:05.353948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:05.353896    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" podStartSLOduration=2.353880693 podStartE2EDuration="2.353880693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:05.351229194 +0000 UTC m=+1550.113666101" watchObservedRunningTime="2026-11-02 07:50:05.353880693 +0000 UTC m=+1550.116317606"
Nov 02 07:50:10.601359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.599069    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:50:10.604478 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.604427    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.630677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.630650    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-894bbb\"/\"default-dockercfg-v9zsq\""
Nov 02 07:50:10.631026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.631005    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:50:10.639302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.639250    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-894bbb\"/\"kube-root-ca.crt\""
Nov 02 07:50:10.639302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.639303    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-894bbb\"/\"openshift-service-ca.crt\""
Nov 02 07:50:10.639495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.639327    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-894bbb\"/\"praxis-route-894bbb-kserve-self-signed-certs\""
Nov 02 07:50:10.750287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750253    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.750480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750332    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.750480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750472    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.750554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750504    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqpl7\" (UniqueName: \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.750554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750537    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.750629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.750612    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858135    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858287    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858355    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqpl7\" (UniqueName: \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858405    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858465    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.858554    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.872494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.863711    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.906330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.889667    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.906330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.889928    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.906330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.896640    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.916584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.916541    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.924453 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.922032    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqpl7\" (UniqueName: \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\") pod \"praxis-route-894bbb-kserve-bb6966cc5-f5mgh\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") " pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:10.930633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:10.930601    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:11.300521 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:50:11.300490    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47cea44e_69c0_4763_88d0_cf90e121410d.slice/crio-3ea94fee059c79e7287e247574c41d163b20a9f4ff979504efdcbc2cb602e34a WatchSource:0}: Error finding container 3ea94fee059c79e7287e247574c41d163b20a9f4ff979504efdcbc2cb602e34a: Status 404 returned error can't find the container with id 3ea94fee059c79e7287e247574c41d163b20a9f4ff979504efdcbc2cb602e34a
Nov 02 07:50:11.302543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:11.302520    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:50:11.360880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:11.360844    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" event={"ID":"47cea44e-69c0-4763-88d0-cf90e121410d","Type":"ContainerStarted","Data":"3ea94fee059c79e7287e247574c41d163b20a9f4ff979504efdcbc2cb602e34a"}
Nov 02 07:50:12.369108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.369077    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" event={"ID":"47cea44e-69c0-4763-88d0-cf90e121410d","Type":"ContainerStarted","Data":"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"}
Nov 02 07:50:12.414518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.414461    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podStartSLOduration=2.414448606 podStartE2EDuration="2.414448606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:12.412913343 +0000 UTC m=+1557.175350249" watchObservedRunningTime="2026-11-02 07:50:12.414448606 +0000 UTC m=+1557.176885512"
Nov 02 07:50:12.500907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.500867    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:50:12.501156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.501129    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" podUID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" containerName="maas-api" containerID="cri-o://eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d" gracePeriod=30
Nov 02 07:50:12.866334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.865634    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:50:12.941459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:12.940274    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:13.072199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.072162    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\") pod \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") "
Nov 02 07:50:13.072789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.072761    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\") pod \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") "
Nov 02 07:50:13.073059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.073033    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\") pod \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") "
Nov 02 07:50:13.073259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.073242    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\" (UniqueName: \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\") pod \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\" (UID: \"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111\") "
Nov 02 07:50:13.074884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.074854    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca" pod "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" (UID: "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:13.078887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.078865    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs" pod "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" (UID: "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:13.079020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.078902    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls" pod "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" (UID: "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:13.079110 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.078910    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp" pod "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" (UID: "19cc7094-c2c8-4a6b-b8c4-a0cd868a5111"). InnerVolumeSpecName "kube-api-access-kkchp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:13.175767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.175728    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkchp\" (UniqueName: \"kubernetes.io/projected/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-kube-api-access-kkchp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:13.175767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.175760    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:13.175767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.175770    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:13.175767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.175778    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:13.377322 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.377282    4515 generic.go:356] "Generic (PLEG): container finished" podID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" containerID="eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d" exitCode=0
Nov 02 07:50:13.377855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.377338    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"
Nov 02 07:50:13.377855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.377366    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" event={"ID":"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111","Type":"ContainerDied","Data":"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"}
Nov 02 07:50:13.377855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.377416    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6" event={"ID":"19cc7094-c2c8-4a6b-b8c4-a0cd868a5111","Type":"ContainerDied","Data":"a7287823935fa6b44e51fbdeb52412e98c0e1954269b66efe39f2a4dd7d7ba5d"}
Nov 02 07:50:13.377855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.377441    4515 scope.go:122] "RemoveContainer" containerID="eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"
Nov 02 07:50:13.388506 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.388483    4515 scope.go:122] "RemoveContainer" containerID="eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"
Nov 02 07:50:13.388774 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:13.388757    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d\": container with ID starting with eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d not found: ID does not exist" containerID="eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"
Nov 02 07:50:13.388828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.388782    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d"} err="failed to get container status \"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d\": rpc error: code = NotFound desc = could not find container \"eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d\": container with ID starting with eca0211672ae889657455933d2638a8cce33348f4cba78a6bb6c866bd68fe71d not found: ID does not exist"
Nov 02 07:50:13.479627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.479093    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:50:13.483141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:13.483113    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b2f76f-54c997497f-t69z6"]
Nov 02 07:50:15.374302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:15.374268    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" path="/var/lib/kubelet/pods/19cc7094-c2c8-4a6b-b8c4-a0cd868a5111/volumes"
Nov 02 07:50:16.326780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:16.326749    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:16.332152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:16.332116    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:16.791671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:16.791638    4515 scope.go:122] "RemoveContainer" containerID="cc672a5e27e2aef22fa00be2384d8177b03b1a385fc8a12c385e9c1d5f7c6b47"
Nov 02 07:50:17.724710 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:17.724669    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:50:20.932092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:20.932058    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:20.946376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:20.946348    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:20.991743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:20.991712    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwdh9_dad9dc47-790e-4c20-952e-c2984691c5d4/payload-processing/0.log"
Nov 02 07:50:21.196066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:21.195985    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwdh9_dad9dc47-790e-4c20-952e-c2984691c5d4/payload-processing/0.log"
Nov 02 07:50:21.421607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:21.421577    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd_075f8a70-0edc-42f3-9f1e-5f72b059f5bc/payload-processing/0.log"
Nov 02 07:50:21.506152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:21.506116    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:50:23.585740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.585701    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:50:23.586379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.586359    4515 cpu_manager.go:414] "containerMap: removing container" podUID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" containerName="maas-api"
Nov 02 07:50:23.586379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.586377    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" containerName="maas-api"
Nov 02 07:50:23.586619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.586607    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cc7094-c2c8-4a6b-b8c4-a0cd868a5111" containerName="maas-api"
Nov 02 07:50:23.592399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.592365    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.623501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.615676    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:50:23.623501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.621548    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.625475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.625447    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:50:23.643553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.643526    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:50:23.716514 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716481    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.716700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716525    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.716700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716605    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.716700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716656    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.716700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716686    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.716835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716732    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.716835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716778    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.716835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716818    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.716931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716852    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.716931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716883    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58zh\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.716931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716908    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.717038 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.716981    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.717072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717032    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.717107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717070    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.717172 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717159    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.717210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717183    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.717210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717203    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.717273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.717227    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq4q\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823188    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.823496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823241    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823272    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq4q\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823422    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823459    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823514    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823563    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823600    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823624    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823679    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823705    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823730    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.823784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823775    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58zh\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.824211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823813    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.824211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.823850    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.824211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.824049    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.824211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.824120    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.835336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.834543    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.841303    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.842141    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.843628    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.843873    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.844103    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.845622    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.846485    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.846724    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.847502    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.849386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.848155    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.850906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.850880    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.854128 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.854085    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.868339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.861022    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.880515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.880437    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58zh\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.883127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.883066    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:23.888028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.887997    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq4q\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.892983 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.892878    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.926349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.922675    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:23.951558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:23.950845    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:24.356337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.339516    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:50:24.444341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.405456    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:50:24.464297 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:50:24.431093    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d452c5_0a3d_4e6a_ae4a_f9166a6fe80d.slice/crio-7fb9a28ccef2c828b072d4cdb27b7069fb2ca4ceae585ea3ed92fa4c41d5cb91 WatchSource:0}: Error finding container 7fb9a28ccef2c828b072d4cdb27b7069fb2ca4ceae585ea3ed92fa4c41d5cb91: Status 404 returned error can't find the container with id 7fb9a28ccef2c828b072d4cdb27b7069fb2ca4ceae585ea3ed92fa4c41d5cb91
Nov 02 07:50:24.464521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.445711    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.464708 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.464681    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.464848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.464830    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.468561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.463204    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.468561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.465402    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.468561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.465453    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.526324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.525539    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" event={"ID":"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d","Type":"ContainerStarted","Data":"7fb9a28ccef2c828b072d4cdb27b7069fb2ca4ceae585ea3ed92fa4c41d5cb91"}
Nov 02 07:50:24.527334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:24.527248    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" event={"ID":"12f7a25a-f059-4817-a209-e91edf3115b0","Type":"ContainerStarted","Data":"cdd06dea7aa87fb6f7ba5e621ad94a8739e6b03b9399b36288a69336c592ad45"}
Nov 02 07:50:25.631300 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.631255    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" event={"ID":"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d","Type":"ContainerStarted","Data":"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"}
Nov 02 07:50:25.640426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.640386    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" event={"ID":"12f7a25a-f059-4817-a209-e91edf3115b0","Type":"ContainerStarted","Data":"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"}
Nov 02 07:50:25.736335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.716625    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podStartSLOduration=2.716602538 podStartE2EDuration="2.716602538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:25.709097162 +0000 UTC m=+1570.471534088" watchObservedRunningTime="2026-11-02 07:50:25.716602538 +0000 UTC m=+1570.479039445"
Nov 02 07:50:25.760024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.759959    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podStartSLOduration=2.7599386 podStartE2EDuration="2.7599386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:25.757771912 +0000 UTC m=+1570.520208837" watchObservedRunningTime="2026-11-02 07:50:25.7599386 +0000 UTC m=+1570.522375507"
Nov 02 07:50:25.935865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.935326    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:25.935865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.935472    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:25.935865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.935518    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:25.967224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.967187    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:25.967631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.967604    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:25.967801 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:25.967775    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:26.255662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.255205    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:50:26.255662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.255262    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:26.577400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.561436    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:26.680026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.662383    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:26.697968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.662756    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:26.698481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.667296    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:26.937390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.923807    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:26.937390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.923884    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:26.973380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.961729    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:26.973380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:26.961802    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:27.953393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:27.939623    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:27.953393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:27.939685    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:27.979353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:27.957720    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:27.979353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:27.957790    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:28.925772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:28.925724    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:28.926205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:28.926173    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:28.962015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:28.961968    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:28.962798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:28.962747    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:29.944334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:29.943483    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:29.944334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:29.943573    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:29.965335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:29.957734    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:29.965335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:29.957797    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:30.937334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:30.928697    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:30.937334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:30.928769    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:30.967331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:30.966479    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:30.967331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:30.966544    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:31.938069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:31.935703    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:50:31.938069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:31.935770    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 07:50:31.958331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:31.953712    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 07:50:31.958331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:31.953779    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 07:50:32.959274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:32.959242    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:32.959905 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:32.959883    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:32.961046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:32.961022    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:50:33.125380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:33.115538    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:33.125380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:33.116480    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:33.125380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:33.120402    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:50:35.450407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.450284    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:50:35.450952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.450570    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" podUID="db23ab4f-340f-45b8-97d0-25333979675d" containerName="maas-api" containerID="cri-o://1013cafd25dc505611a6e57add65533491cb628f1245ef54f0d0d81f837a2aee" gracePeriod=30
Nov 02 07:50:35.712898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.712822    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:50:35.713122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.713093    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" podUID="172a4882-20f2-4f7a-a34b-b35edee77fd5" containerName="payload-pre-processing" containerID="cri-o://ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b" gracePeriod=30
Nov 02 07:50:35.804487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.804450    4515 generic.go:356] "Generic (PLEG): container finished" podID="db23ab4f-340f-45b8-97d0-25333979675d" containerID="1013cafd25dc505611a6e57add65533491cb628f1245ef54f0d0d81f837a2aee" exitCode=0
Nov 02 07:50:35.804741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.804526    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" event={"ID":"db23ab4f-340f-45b8-97d0-25333979675d","Type":"ContainerDied","Data":"1013cafd25dc505611a6e57add65533491cb628f1245ef54f0d0d81f837a2aee"}
Nov 02 07:50:35.804741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.804591    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz" event={"ID":"db23ab4f-340f-45b8-97d0-25333979675d","Type":"ContainerDied","Data":"1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6"}
Nov 02 07:50:35.804741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.804620    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fbd0977817be83e7b1580c887aaa5753307d7ad780dd444921781f4d3ea31f6"
Nov 02 07:50:35.806497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.806470    4515 generic.go:356] "Generic (PLEG): container finished" podID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerID="8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a" exitCode=0
Nov 02 07:50:35.806625 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.806520    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerDied","Data":"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"}
Nov 02 07:50:35.806987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.806966    4515 scope.go:122] "RemoveContainer" containerID="8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"
Nov 02 07:50:35.808674 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.808644    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:35.808751 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.808716    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:35.808811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.808760    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:35.809867 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.809852    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:50:35.913882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.913844    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\" (UniqueName: \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\") pod \"db23ab4f-340f-45b8-97d0-25333979675d\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") "
Nov 02 07:50:35.914071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.913968    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\") pod \"db23ab4f-340f-45b8-97d0-25333979675d\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") "
Nov 02 07:50:35.914071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.914030    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\") pod \"db23ab4f-340f-45b8-97d0-25333979675d\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") "
Nov 02 07:50:35.914226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.914148    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") pod \"db23ab4f-340f-45b8-97d0-25333979675d\" (UID: \"db23ab4f-340f-45b8-97d0-25333979675d\") "
Nov 02 07:50:35.918521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.918468    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca" pod "db23ab4f-340f-45b8-97d0-25333979675d" (UID: "db23ab4f-340f-45b8-97d0-25333979675d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:35.919150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.918670    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p" pod "db23ab4f-340f-45b8-97d0-25333979675d" (UID: "db23ab4f-340f-45b8-97d0-25333979675d"). InnerVolumeSpecName "kube-api-access-px57p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:35.920002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.919974    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs" pod "db23ab4f-340f-45b8-97d0-25333979675d" (UID: "db23ab4f-340f-45b8-97d0-25333979675d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:35.920187 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:35.920167    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls" pod "db23ab4f-340f-45b8-97d0-25333979675d" (UID: "db23ab4f-340f-45b8-97d0-25333979675d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:36.016721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.016625    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db23ab4f-340f-45b8-97d0-25333979675d-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.016721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.016664    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.016721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.016678    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db23ab4f-340f-45b8-97d0-25333979675d-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.016721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.016691    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px57p\" (UniqueName: \"kubernetes.io/projected/db23ab4f-340f-45b8-97d0-25333979675d-kube-api-access-px57p\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.262257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.248768    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:50:36.269072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.268961    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\") pod \"172a4882-20f2-4f7a-a34b-b35edee77fd5\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") "
Nov 02 07:50:36.269072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.269042    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\" (UniqueName: \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\") pod \"172a4882-20f2-4f7a-a34b-b35edee77fd5\" (UID: \"172a4882-20f2-4f7a-a34b-b35edee77fd5\") "
Nov 02 07:50:36.270533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.270472    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume" pod "172a4882-20f2-4f7a-a34b-b35edee77fd5" (UID: "172a4882-20f2-4f7a-a34b-b35edee77fd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:36.272883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.272860    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs" pod "172a4882-20f2-4f7a-a34b-b35edee77fd5" (UID: "172a4882-20f2-4f7a-a34b-b35edee77fd5"). InnerVolumeSpecName "kube-api-access-jpbgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:36.370915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.370872    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172a4882-20f2-4f7a-a34b-b35edee77fd5-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.370915 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.370899    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbgs\" (UniqueName: \"kubernetes.io/projected/172a4882-20f2-4f7a-a34b-b35edee77fd5-kube-api-access-jpbgs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.815293 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.815253    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerStarted","Data":"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"}
Nov 02 07:50:36.816874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.816844    4515 generic.go:356] "Generic (PLEG): container finished" podID="172a4882-20f2-4f7a-a34b-b35edee77fd5" containerID="ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b" exitCode=0
Nov 02 07:50:36.820552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.820531    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"
Nov 02 07:50:36.824378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.817147    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" event={"ID":"172a4882-20f2-4f7a-a34b-b35edee77fd5","Type":"ContainerDied","Data":"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"}
Nov 02 07:50:36.824638 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.824614    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb" event={"ID":"172a4882-20f2-4f7a-a34b-b35edee77fd5","Type":"ContainerDied","Data":"96e633b340ab956bcf6a35b25978f7fd646513a7d19d28572bbdc976e398723e"}
Nov 02 07:50:36.824756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.824742    4515 scope.go:122] "RemoveContainer" containerID="ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"
Nov 02 07:50:36.825010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.817128    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"
Nov 02 07:50:36.855325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.855293    4515 scope.go:122] "RemoveContainer" containerID="ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"
Nov 02 07:50:36.855687 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:36.855666    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b\": container with ID starting with ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b not found: ID does not exist" containerID="ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"
Nov 02 07:50:36.855763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.855697    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b"} err="failed to get container status \"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b\": rpc error: code = NotFound desc = could not find container \"ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b\": container with ID starting with ed3affce42d9084274fbc235e5b93fcefda02caae62cac2018c80d331f80288b not found: ID does not exist"
Nov 02 07:50:36.912704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.912669    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:50:36.940349 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.937783    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f3716-79db7bb5c9-dbdmb"]
Nov 02 07:50:36.963676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.963644    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:50:36.978902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:36.978866    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f3716-5765b68fb9-dcdtz"]
Nov 02 07:50:37.381579 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:37.381544    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172a4882-20f2-4f7a-a34b-b35edee77fd5" path="/var/lib/kubelet/pods/172a4882-20f2-4f7a-a34b-b35edee77fd5/volumes"
Nov 02 07:50:37.382054 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:37.382039    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db23ab4f-340f-45b8-97d0-25333979675d" path="/var/lib/kubelet/pods/db23ab4f-340f-45b8-97d0-25333979675d/volumes"
Nov 02 07:50:37.439717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:37.439678    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:50:37.442378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:37.442345    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:50:37.442553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:37.442415    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 02 07:50:38.441802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:38.441759    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:50:38.442431 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:38.442401    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 02 07:50:38.763601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:38.763570    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:50:39.441735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:39.441693    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:50:39.441959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:39.441756    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 02 07:50:40.450335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:40.439868    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:50:40.450335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:40.439929    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 02 07:50:41.439335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:41.439140    4515 patch_prober.go:29] interesting pod/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:50:41.439335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:41.439205    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 02 07:50:42.079197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.079158    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:50:42.085733 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.085699    4515 cpu_manager.go:414] "containerMap: removing container" podUID="db23ab4f-340f-45b8-97d0-25333979675d" containerName="maas-api"
Nov 02 07:50:42.085987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.085969    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db23ab4f-340f-45b8-97d0-25333979675d" containerName="maas-api"
Nov 02 07:50:42.086174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.086162    4515 cpu_manager.go:414] "containerMap: removing container" podUID="172a4882-20f2-4f7a-a34b-b35edee77fd5" containerName="payload-pre-processing"
Nov 02 07:50:42.086264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.086251    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172a4882-20f2-4f7a-a34b-b35edee77fd5" containerName="payload-pre-processing"
Nov 02 07:50:42.086862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.086836    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="db23ab4f-340f-45b8-97d0-25333979675d" containerName="maas-api"
Nov 02 07:50:42.087046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.087033    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="172a4882-20f2-4f7a-a34b-b35edee77fd5" containerName="payload-pre-processing"
Nov 02 07:50:42.099131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.099101    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.113394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.112073    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:50:42.144345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.142019    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-677726-dockercfg-w7llt\""
Nov 02 07:50:42.144345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.142792    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-677726\""
Nov 02 07:50:42.202014 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.201972    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d8fw\" (UniqueName: \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.202567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.202541    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.304239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.304201    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.304445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.304277    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d8fw\" (UniqueName: \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.305353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.305297    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.328165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.328131    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d8fw\" (UniqueName: \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\") pod \"payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.511549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.511514    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:42.664984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.653920    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:50:42.664984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.655704    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:50:42.825047 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:50:42.825014    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009650f_2c70_4f4e_bea4_77518b0c5b12.slice/crio-da036b7ffa49568c694eeff2bfc677e0aa0b4e517bb1cdc68d91fd41b81387ae WatchSource:0}: Error finding container da036b7ffa49568c694eeff2bfc677e0aa0b4e517bb1cdc68d91fd41b81387ae: Status 404 returned error can't find the container with id da036b7ffa49568c694eeff2bfc677e0aa0b4e517bb1cdc68d91fd41b81387ae
Nov 02 07:50:42.825569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.825546    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:50:42.986058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.986013    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" event={"ID":"7009650f-2c70-4f4e-bea4-77518b0c5b12","Type":"ContainerStarted","Data":"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"}
Nov 02 07:50:42.986255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:42.986134    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" event={"ID":"7009650f-2c70-4f4e-bea4-77518b0c5b12","Type":"ContainerStarted","Data":"da036b7ffa49568c694eeff2bfc677e0aa0b4e517bb1cdc68d91fd41b81387ae"}
Nov 02 07:50:43.030526 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:43.030477    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" podStartSLOduration=1.03046209 podStartE2EDuration="1.03046209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:43.028548496 +0000 UTC m=+1587.790985404" watchObservedRunningTime="2026-11-02 07:50:43.03046209 +0000 UTC m=+1587.792899014"
Nov 02 07:50:43.992051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:43.992014    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:45.548934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:45.548891    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:50:45.549490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:45.549165    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy" containerID="cri-o://808c39ccb8fab7e9c86b7c0b429f2322dd86339cfcfb699b3ed6a79199e110e1" gracePeriod=30
Nov 02 07:50:47.724233 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:47.724198    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:50:48.955814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:48.955782    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:50:51.112335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.111526    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:50:51.122342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.117851    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:50:51.131793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.131757    4515 generic.go:356] "Generic (PLEG): container finished" podID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerID="808c39ccb8fab7e9c86b7c0b429f2322dd86339cfcfb699b3ed6a79199e110e1" exitCode=0
Nov 02 07:50:51.239901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.199389    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerDied","Data":"808c39ccb8fab7e9c86b7c0b429f2322dd86339cfcfb699b3ed6a79199e110e1"}
Nov 02 07:50:51.239901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.199478    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6" event={"ID":"3cf760a0-0c5d-4292-9163-cb69e70d2444","Type":"ContainerDied","Data":"2f99daf47315026e09b27ab808244bc8522a0782de8b209b418988b1c76a2087"}
Nov 02 07:50:51.239901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.199505    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f99daf47315026e09b27ab808244bc8522a0782de8b209b418988b1c76a2087"
Nov 02 07:50:51.239901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.199527    4515 scope.go:122] "RemoveContainer" containerID="88f39b68defefd59593caaa0021922e29e7207f28f1f71f5da8a5bba8c77113e"
Nov 02 07:50:51.239901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.199871    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy" containerID="cri-o://6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f" gracePeriod=30
Nov 02 07:50:51.265822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.259257    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.314583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.314548    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.451259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.451219    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.500708    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.500791    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.500943    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.501067    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.501178    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.532249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.501295    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\") pod \"3cf760a0-0c5d-4292-9163-cb69e70d2444\" (UID: \"3cf760a0-0c5d-4292-9163-cb69e70d2444\") "
Nov 02 07:50:51.533346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.533201    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:51.535328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.535092    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:51.543491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.535563    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:51.564256 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.543747    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:51.638028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.611952    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf760a0-0c5d-4292-9163-cb69e70d2444-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.659613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.659567    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.659879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.659862    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.660022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.660007    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.714334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.660892    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:51.755614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.754482    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:51.775338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.771453    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.775338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.771488    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.818690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.818604    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "kube-api-access-s44l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:51.886648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.886604    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44l9\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-kube-api-access-s44l9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:51.972345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.967525    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:51.972345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.969149    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:51.988691 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:51.988637    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:52.023641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:52.020584    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token" pod "3cf760a0-0c5d-4292-9163-cb69e70d2444" (UID: "3cf760a0-0c5d-4292-9163-cb69e70d2444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:52.092651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:52.092603    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf760a0-0c5d-4292-9163-cb69e70d2444-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:52.171686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:52.171658    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"
Nov 02 07:50:52.281428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:52.281383    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:50:52.289331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:52.288953    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f3716-openshift-default-67dfcf5dcf-xvsg6"]
Nov 02 07:50:53.375020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.374986    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" path="/var/lib/kubelet/pods/3cf760a0-0c5d-4292-9163-cb69e70d2444/volumes"
Nov 02 07:50:53.518935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.518898    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:50:53.519177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.519150    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerName="maas-api" containerID="cri-o://3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d" gracePeriod=30
Nov 02 07:50:53.653658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.653610    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.156:8443/health\": dial tcp 10.133.0.156:8443: connect: connection refused"
Nov 02 07:50:53.769118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.769074    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:53.769530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.769503    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" podUID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" containerName="payload-processing" containerID="cri-o://2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0" gracePeriod=30
Nov 02 07:50:53.794353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.794320    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:53.794605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.794558    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" podUID="1742c837-da1e-4fed-b681-c0f2d7aeb385" containerName="payload-pre-processing" containerID="cri-o://f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd" gracePeriod=30
Nov 02 07:50:53.939900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:53.939179    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:50:54.055464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.055433    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\" (UniqueName: \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\") pod \"79442b48-e4b1-4704-a794-1c4077b9f5b9\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") "
Nov 02 07:50:54.055627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.055595    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\") pod \"79442b48-e4b1-4704-a794-1c4077b9f5b9\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") "
Nov 02 07:50:54.055714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.055642    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\") pod \"79442b48-e4b1-4704-a794-1c4077b9f5b9\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") "
Nov 02 07:50:54.055772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.055726    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\") pod \"79442b48-e4b1-4704-a794-1c4077b9f5b9\" (UID: \"79442b48-e4b1-4704-a794-1c4077b9f5b9\") "
Nov 02 07:50:54.056013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.055986    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca" pod "79442b48-e4b1-4704-a794-1c4077b9f5b9" (UID: "79442b48-e4b1-4704-a794-1c4077b9f5b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:54.059644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.059617    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs" pod "79442b48-e4b1-4704-a794-1c4077b9f5b9" (UID: "79442b48-e4b1-4704-a794-1c4077b9f5b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:54.059770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.059725    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r" pod "79442b48-e4b1-4704-a794-1c4077b9f5b9" (UID: "79442b48-e4b1-4704-a794-1c4077b9f5b9"). InnerVolumeSpecName "kube-api-access-nn87r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:54.061166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.061144    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls" pod "79442b48-e4b1-4704-a794-1c4077b9f5b9" (UID: "79442b48-e4b1-4704-a794-1c4077b9f5b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:54.062497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.062476    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79442b48-e4b1-4704-a794-1c4077b9f5b9-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.062582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.062503    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.062582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.062518    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79442b48-e4b1-4704-a794-1c4077b9f5b9-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.062582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.062532    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn87r\" (UniqueName: \"kubernetes.io/projected/79442b48-e4b1-4704-a794-1c4077b9f5b9-kube-api-access-nn87r\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.084970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.084942    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:54.102184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.102151    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:54.176521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.176294    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\" (UniqueName: \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\") pod \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") "
Nov 02 07:50:54.176521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.176433    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\") pod \"1742c837-da1e-4fed-b681-c0f2d7aeb385\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") "
Nov 02 07:50:54.176521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.176476    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\") pod \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\" (UID: \"075f8a70-0edc-42f3-9f1e-5f72b059f5bc\") "
Nov 02 07:50:54.176893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.176650    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\") pod \"1742c837-da1e-4fed-b681-c0f2d7aeb385\" (UID: \"1742c837-da1e-4fed-b681-c0f2d7aeb385\") "
Nov 02 07:50:54.208376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.195698    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume" pod "075f8a70-0edc-42f3-9f1e-5f72b059f5bc" (UID: "075f8a70-0edc-42f3-9f1e-5f72b059f5bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:54.227648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.227609    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume" pod "1742c837-da1e-4fed-b681-c0f2d7aeb385" (UID: "1742c837-da1e-4fed-b681-c0f2d7aeb385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.249371    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj" pod "075f8a70-0edc-42f3-9f1e-5f72b059f5bc" (UID: "075f8a70-0edc-42f3-9f1e-5f72b059f5bc"). InnerVolumeSpecName "kube-api-access-bx5dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.249561    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r" pod "1742c837-da1e-4fed-b681-c0f2d7aeb385" (UID: "1742c837-da1e-4fed-b681-c0f2d7aeb385"). InnerVolumeSpecName "kube-api-access-2p82r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.282566    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/1742c837-da1e-4fed-b681-c0f2d7aeb385-kube-api-access-2p82r\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.282603    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5dj\" (UniqueName: \"kubernetes.io/projected/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-kube-api-access-bx5dj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.282617    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1742c837-da1e-4fed-b681-c0f2d7aeb385-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.299654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.282630    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075f8a70-0edc-42f3-9f1e-5f72b059f5bc-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:54.321696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.321657    4515 generic.go:356] "Generic (PLEG): container finished" podID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerID="3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d" exitCode=0
Nov 02 07:50:54.321861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.321761    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" event={"ID":"79442b48-e4b1-4704-a794-1c4077b9f5b9","Type":"ContainerDied","Data":"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"}
Nov 02 07:50:54.321861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.321811    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4" event={"ID":"79442b48-e4b1-4704-a794-1c4077b9f5b9","Type":"ContainerDied","Data":"a235b2d43b8f13ea6c204f989ebd1759f0250050d7474c68a2ecfe6816a673cd"}
Nov 02 07:50:54.321861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.321842    4515 scope.go:122] "RemoveContainer" containerID="3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"
Nov 02 07:50:54.322043 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.321971    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"
Nov 02 07:50:54.326947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.326711    4515 generic.go:356] "Generic (PLEG): container finished" podID="1742c837-da1e-4fed-b681-c0f2d7aeb385" containerID="f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd" exitCode=0
Nov 02 07:50:54.326947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.326786    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"
Nov 02 07:50:54.326947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.326867    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" event={"ID":"1742c837-da1e-4fed-b681-c0f2d7aeb385","Type":"ContainerDied","Data":"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"}
Nov 02 07:50:54.326947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.326913    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz" event={"ID":"1742c837-da1e-4fed-b681-c0f2d7aeb385","Type":"ContainerDied","Data":"b1a75501b6741b1d1ade4f7704e70944cd9f3fcf96a368b076e445ade0780ede"}
Nov 02 07:50:54.328882 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.328853    4515 generic.go:356] "Generic (PLEG): container finished" podID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" containerID="2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0" exitCode=0
Nov 02 07:50:54.328990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.328975    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" event={"ID":"075f8a70-0edc-42f3-9f1e-5f72b059f5bc","Type":"ContainerDied","Data":"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"}
Nov 02 07:50:54.329064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.329046    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd" event={"ID":"075f8a70-0edc-42f3-9f1e-5f72b059f5bc","Type":"ContainerDied","Data":"4630fa4a17bad115167f7f75a777dd9f2a5f78ee4665a87d388579db59d3932e"}
Nov 02 07:50:54.329143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.329130    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"
Nov 02 07:50:54.342865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.342840    4515 scope.go:122] "RemoveContainer" containerID="3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"
Nov 02 07:50:54.343214 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:54.343195    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d\": container with ID starting with 3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d not found: ID does not exist" containerID="3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"
Nov 02 07:50:54.343363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.343227    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d"} err="failed to get container status \"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d\": rpc error: code = NotFound desc = could not find container \"3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d\": container with ID starting with 3eacca6f7b31ed0fc6360db688921d006d8cd237fe1d0e1d4ac896f29425395d not found: ID does not exist"
Nov 02 07:50:54.343363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.343251    4515 scope.go:122] "RemoveContainer" containerID="f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"
Nov 02 07:50:54.353701 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.353681    4515 scope.go:122] "RemoveContainer" containerID="f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"
Nov 02 07:50:54.354015 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:54.353995    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd\": container with ID starting with f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd not found: ID does not exist" containerID="f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"
Nov 02 07:50:54.354072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.354025    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd"} err="failed to get container status \"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd\": rpc error: code = NotFound desc = could not find container \"f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd\": container with ID starting with f3831f15c012369b99019b21227d76e37bde38139f33ed6024e5dc67b7dda2fd not found: ID does not exist"
Nov 02 07:50:54.354072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.354044    4515 scope.go:122] "RemoveContainer" containerID="2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"
Nov 02 07:50:54.363396 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.363378    4515 scope.go:122] "RemoveContainer" containerID="2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"
Nov 02 07:50:54.363664 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:54.363646    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0\": container with ID starting with 2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0 not found: ID does not exist" containerID="2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"
Nov 02 07:50:54.363704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.363671    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0"} err="failed to get container status \"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0\": rpc error: code = NotFound desc = could not find container \"2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0\": container with ID starting with 2dd547458626e1e443d81b3a2d4e3df7848372b54191a51d6e8e14d60d229ea0 not found: ID does not exist"
Nov 02 07:50:54.378521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.378498    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:50:54.384918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.384896    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb64ad-5cdf79ffb4-vzbr4"]
Nov 02 07:50:54.428566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.428490    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:54.431253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.431232    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-7cf5b9c887-4llfz"]
Nov 02 07:50:54.488201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.488160    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:54.495559 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:54.495529    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-6db8db6f44-ghlhd"]
Nov 02 07:50:55.000979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:55.000953    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:50:55.374473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:55.374399    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" path="/var/lib/kubelet/pods/075f8a70-0edc-42f3-9f1e-5f72b059f5bc/volumes"
Nov 02 07:50:55.374821 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:55.374807    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1742c837-da1e-4fed-b681-c0f2d7aeb385" path="/var/lib/kubelet/pods/1742c837-da1e-4fed-b681-c0f2d7aeb385/volumes"
Nov 02 07:50:55.375204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:55.375191    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" path="/var/lib/kubelet/pods/79442b48-e4b1-4704-a794-1c4077b9f5b9/volumes"
Nov 02 07:50:56.610378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.610351    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:50:56.618824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.618795    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.618996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.618864    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.618996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.618932    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.618996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.618955    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.618996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.618977    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619027    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619063    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619076    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619107    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619121    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.619213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619175    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\") pod \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\" (UID: \"240ac0d5-3c3e-4205-9dbb-48ee9f90d121\") "
Nov 02 07:50:56.619537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619291    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:56.619537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619410    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.619537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619514    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.619887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619860    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.619887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619888    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.620083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619903    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.620083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619916    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.620083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.619931    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.621454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.621429    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "kube-api-access-kqhdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:56.621547 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.621521    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:56.621713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.621692    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:56.621971 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.621946    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy" pod "240ac0d5-3c3e-4205-9dbb-48ee9f90d121" (UID: "240ac0d5-3c3e-4205-9dbb-48ee9f90d121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.721891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.721807    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.721891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.721837    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.721891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.721850    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqhdk\" (UniqueName: \"kubernetes.io/projected/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-kube-api-access-kqhdk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.721891 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:56.721864    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240ac0d5-3c3e-4205-9dbb-48ee9f90d121-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:50:57.355771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.355729    4515 generic.go:356] "Generic (PLEG): container finished" podID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerID="6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f" exitCode=0
Nov 02 07:50:57.355771 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.355760    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerDied","Data":"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"}
Nov 02 07:50:57.355982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.355787    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"
Nov 02 07:50:57.355982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.355851    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2" event={"ID":"240ac0d5-3c3e-4205-9dbb-48ee9f90d121","Type":"ContainerDied","Data":"89b35631b6db791090997efdcdb2d04656880fce22ce81042c1f77cfa12079a7"}
Nov 02 07:50:57.355982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.355877    4515 scope.go:122] "RemoveContainer" containerID="6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"
Nov 02 07:50:57.367271 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.367248    4515 scope.go:122] "RemoveContainer" containerID="cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"
Nov 02 07:50:57.377684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.377664    4515 scope.go:122] "RemoveContainer" containerID="6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"
Nov 02 07:50:57.377988 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:57.377968    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f\": container with ID starting with 6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f not found: ID does not exist" containerID="6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"
Nov 02 07:50:57.378051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.377997    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f"} err="failed to get container status \"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f\": rpc error: code = NotFound desc = could not find container \"6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f\": container with ID starting with 6470ece3b9c2a5bfb58de8c9a2eaf1d7687ba8176005b6dce2d8976ae214e05f not found: ID does not exist"
Nov 02 07:50:57.378051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.378018    4515 scope.go:122] "RemoveContainer" containerID="cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"
Nov 02 07:50:57.378282 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:50:57.378262    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed\": container with ID starting with cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed not found: ID does not exist" containerID="cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"
Nov 02 07:50:57.378372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.378294    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed"} err="failed to get container status \"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed\": rpc error: code = NotFound desc = could not find container \"cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed\": container with ID starting with cc96359797562752d72016f39a1a968aaf939e69a030170858c450ddb99361ed not found: ID does not exist"
Nov 02 07:50:57.425119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.425077    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:50:57.437801 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:57.437774    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-xxtg2"]
Nov 02 07:50:59.376083 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:50:59.376038    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" path="/var/lib/kubelet/pods/240ac0d5-3c3e-4205-9dbb-48ee9f90d121/volumes"
Nov 02 07:51:00.707366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.707323    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:51:00.708096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708073    4515 cpu_manager.go:414] "containerMap: removing container" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerName="maas-api"
Nov 02 07:51:00.708145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708100    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerName="maas-api"
Nov 02 07:51:00.708145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708140    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:00.708211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708149    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:00.708211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708164    4515 cpu_manager.go:414] "containerMap: removing container" podUID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" containerName="payload-processing"
Nov 02 07:51:00.708211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708172    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" containerName="payload-processing"
Nov 02 07:51:00.708211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708195    4515 cpu_manager.go:414] "containerMap: removing container" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:00.708211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708203    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708225    4515 cpu_manager.go:414] "containerMap: removing container" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708232    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708251    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1742c837-da1e-4fed-b681-c0f2d7aeb385" containerName="payload-pre-processing"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708259    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1742c837-da1e-4fed-b681-c0f2d7aeb385" containerName="payload-pre-processing"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708385    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708400    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="79442b48-e4b1-4704-a794-1c4077b9f5b9" containerName="maas-api"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708413    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708420    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="075f8a70-0edc-42f3-9f1e-5f72b059f5bc" containerName="payload-processing"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708429    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:00.708474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.708436    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1742c837-da1e-4fed-b681-c0f2d7aeb385" containerName="payload-pre-processing"
Nov 02 07:51:00.717699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.717676    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.739879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.739849    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:51:00.750808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.750778    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-16e21e\"/\"openshift-service-ca.crt\""
Nov 02 07:51:00.750991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.750839    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-16e21e\"/\"kube-root-ca.crt\""
Nov 02 07:51:00.751957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.751935    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-16e21e\"/\"default-dockercfg-rpqwl\""
Nov 02 07:51:00.752099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.752023    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-16e21e\"/\"ipp-route-16e21e-kserve-self-signed-certs\""
Nov 02 07:51:00.776496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776462    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.776839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776519    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.776839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776600    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.776839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776638    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7fpn\" (UniqueName: \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.776839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776669    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.777164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.776972    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.877831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.877800    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.877941    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.877992    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878031    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878086    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878142    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7fpn\" (UniqueName: \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878444    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878476    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.878709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.878531    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.881347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.880732    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.888164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.888139    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:00.911485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:00.911455    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7fpn\" (UniqueName: \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\") pod \"ipp-route-16e21e-kserve-5dfd49856f-4fqqt\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") " pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:01.030770 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.030733    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:01.288902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.288823    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 07:51:01.289504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.289486    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:01.289557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.289506    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf760a0-0c5d-4292-9163-cb69e70d2444" containerName="istio-proxy"
Nov 02 07:51:01.289604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.289594    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="240ac0d5-3c3e-4205-9dbb-48ee9f90d121" containerName="istio-proxy"
Nov 02 07:51:01.294127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.294110    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.327445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.327413    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 07:51:01.412962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.412929    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.412982    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413144    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413244 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413193    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413257    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413301    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbddd\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413445    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.413525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.413510    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.482927 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:51:01.482886    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2a67473_c35a_43a7_8e77_bbb66098bbb1.slice/crio-aebaa82fb41a4a8c16165cd09310eca7add981930c55c0ac5399f4c510306c18 WatchSource:0}: Error finding container aebaa82fb41a4a8c16165cd09310eca7add981930c55c0ac5399f4c510306c18: Status 404 returned error can't find the container with id aebaa82fb41a4a8c16165cd09310eca7add981930c55c0ac5399f4c510306c18
Nov 02 07:51:01.484744 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.484706    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:51:01.514931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.514902    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.514943    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515018    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515051    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515076    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515097    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515238    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbddd\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515354    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515410    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515428    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515465    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515556    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515629    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.515822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.515751    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.517367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.517345    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.517815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.517796    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.539883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.539801    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.544608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.544584    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbddd\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.610544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.610506    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:01.942040 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:51:01.942001    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bba617_9ead_4c0a_9bb6_689b8c16de59.slice/crio-4388f655e01fa9121a603080b925e8c21608fcb13f63b1fd601c1c047e8b410d WatchSource:0}: Error finding container 4388f655e01fa9121a603080b925e8c21608fcb13f63b1fd601c1c047e8b410d: Status 404 returned error can't find the container with id 4388f655e01fa9121a603080b925e8c21608fcb13f63b1fd601c1c047e8b410d
Nov 02 07:51:01.943511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.943483    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 07:51:01.944946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.944909    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:01.945020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.944994    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:01.945062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:01.945049    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:02.422200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.422164    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" event={"ID":"f4bba617-9ead-4c0a-9bb6-689b8c16de59","Type":"ContainerStarted","Data":"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"}
Nov 02 07:51:02.422418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.422225    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" event={"ID":"f4bba617-9ead-4c0a-9bb6-689b8c16de59","Type":"ContainerStarted","Data":"4388f655e01fa9121a603080b925e8c21608fcb13f63b1fd601c1c047e8b410d"}
Nov 02 07:51:02.423984 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.423961    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" event={"ID":"b2a67473-c35a-43a7-8e77-bbb66098bbb1","Type":"ContainerStarted","Data":"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"}
Nov 02 07:51:02.424081 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.423991    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" event={"ID":"b2a67473-c35a-43a7-8e77-bbb66098bbb1","Type":"ContainerStarted","Data":"aebaa82fb41a4a8c16165cd09310eca7add981930c55c0ac5399f4c510306c18"}
Nov 02 07:51:02.484668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.484614    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podStartSLOduration=1.484599853 podStartE2EDuration="1.484599853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:02.482001468 +0000 UTC m=+1607.244438375" watchObservedRunningTime="2026-11-02 07:51:02.484599853 +0000 UTC m=+1607.247036838"
Nov 02 07:51:02.571335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.565194    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podStartSLOduration=2.56184346 podStartE2EDuration="2.56184346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:02.538857144 +0000 UTC m=+1607.301294050" watchObservedRunningTime="2026-11-02 07:51:02.56184346 +0000 UTC m=+1607.324280363"
Nov 02 07:51:02.611165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:02.611129    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:03.040448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:03.040413    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:03.615889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:03.615841    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:03.616195 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:03.616152    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:04.613570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:04.613513    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:04.613991 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:04.613596    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:05.695334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:05.636183    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:05.695334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:05.636237    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:05.695334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:05.669398    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:05.985521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:05.985421    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:06.051468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.051408    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:06.555711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.555462    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:06.633333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.627785    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:06.633333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.627845    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:06.633333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.632460    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:51:06.639092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:06.638373    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:07.611218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:07.611183    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:07.611656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:07.611239    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:08.611200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:08.611167    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:08.611473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:08.611219    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:09.611743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:09.611704    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 07:51:09.612267 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:09.611759    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 07:51:11.030864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.030824    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:11.044629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.044600    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:11.653502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.653461    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:51:11.653734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.653515    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:11.836074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.836041    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:11.962631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.962554    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:11.963363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.963340    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:11.964052 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:11.964037    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 07:51:16.344811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.344771    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:51:16.350480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.350453    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.382115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.382082    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:51:16.525685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525645    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.525839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525711    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.525903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.525938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525899    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.525938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525930    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.526036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.525975    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v94\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.526149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.526130    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.526220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.526187    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.526411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.526390    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.627682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627581    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.627682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627626    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.627682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627647    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v94\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627689    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627722    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627777    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627839    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627904    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.627992    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.628058    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.628092    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.628222    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.628305    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.628658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.628639    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.630269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.630245    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.630430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.630410    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.657718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.657689    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.663677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.663641    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v94\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.668650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.668622    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:16.996036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.996004    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:51:16.996141 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:51:16.996007    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c77bbef_ebae_4395_bad7_91ba03f47334.slice/crio-6e7d230f17ad84e174b2a269f7a6406ef3d006724efbd2f045b5f23d1bb29910 WatchSource:0}: Error finding container 6e7d230f17ad84e174b2a269f7a6406ef3d006724efbd2f045b5f23d1bb29910: Status 404 returned error can't find the container with id 6e7d230f17ad84e174b2a269f7a6406ef3d006724efbd2f045b5f23d1bb29910
Nov 02 07:51:16.998908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.998871    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:16.999019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.998940    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:16.999019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:16.998968    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.751465 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:51:17.751426    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:51:17.888489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:17.888440    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" event={"ID":"0c77bbef-ebae-4395-bad7-91ba03f47334","Type":"ContainerStarted","Data":"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"}
Nov 02 07:51:17.888660 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:17.888519    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" event={"ID":"0c77bbef-ebae-4395-bad7-91ba03f47334","Type":"ContainerStarted","Data":"6e7d230f17ad84e174b2a269f7a6406ef3d006724efbd2f045b5f23d1bb29910"}
Nov 02 07:51:17.950044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:17.949977    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podStartSLOduration=1.949960714 podStartE2EDuration="1.949960714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:17.946653785 +0000 UTC m=+1622.709090714" watchObservedRunningTime="2026-11-02 07:51:17.949960714 +0000 UTC m=+1622.712397622"
Nov 02 07:51:18.668834 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:18.668802    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:18.670577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:18.670555    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:18.670668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:18.670602    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:19.680470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:19.680414    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:19.681150 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:19.680502    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:20.669201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:20.669164    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:20.669417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:20.669246    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:21.672879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:21.672759    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:21.673499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:21.673463    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:22.669620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:22.669581    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:22.669802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:22.669637    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:23.669521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:23.669480    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 07:51:23.669987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:23.669550    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 07:51:24.674566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:24.674533    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:24.674950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:24.674934    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:24.675570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:24.675549    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:51:27.380437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.380406    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:51:27.523465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.523434    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:51:27.528783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.528764    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.550239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.550208    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3624d8-serving-cert\""
Nov 02 07:51:27.572353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.571667    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:51:27.587459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.587429    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4jw\" (UniqueName: \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.587668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.587555    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.587668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.587604    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.587668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.587641    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.688909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.688831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4jw\" (UniqueName: \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.689060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.688932    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.689060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.688965    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.689060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.688991    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.689603 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.689582    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.691465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.691445    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.691615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.691594    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.715062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.715032    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4jw\" (UniqueName: \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\") pod \"maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:27.858575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:27.857671    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:28.186585 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:51:28.186551    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d7e05b_e821_495c_9ee2_a83dbdc349eb.slice/crio-fefc73122d40fa0bc16684b68a5259ceecdb79cf5a1717166080c87332e08c5b WatchSource:0}: Error finding container fefc73122d40fa0bc16684b68a5259ceecdb79cf5a1717166080c87332e08c5b: Status 404 returned error can't find the container with id fefc73122d40fa0bc16684b68a5259ceecdb79cf5a1717166080c87332e08c5b
Nov 02 07:51:28.188680 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:28.188654    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:51:29.209339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:29.207675    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" event={"ID":"c5d7e05b-e821-495c-9ee2-a83dbdc349eb","Type":"ContainerStarted","Data":"fefc73122d40fa0bc16684b68a5259ceecdb79cf5a1717166080c87332e08c5b"}
Nov 02 07:51:29.809417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:29.802942    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:51:29.830617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:29.830544    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:29.996465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:29.996410    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.268334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.252376    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" event={"ID":"c5d7e05b-e821-495c-9ee2-a83dbdc349eb","Type":"ContainerStarted","Data":"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"}
Nov 02 07:51:30.268334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.253596    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:30.535402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.532398    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.618344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.615611    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.684466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.684421    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.803494 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.803388    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.804487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.804463    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:51:30.831472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.831388    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": context deadline exceeded"
Nov 02 07:51:30.832582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.832549    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:30.882530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:30.882451    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" podStartSLOduration=3.265915174 podStartE2EDuration="3.882429765s" totalImagesPullingTime="616.514591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:30.317099383 +0000 UTC m=+1635.079536290" watchObservedRunningTime="2026-11-02 07:51:30.882429765 +0000 UTC m=+1635.644866681"
Nov 02 07:51:31.075290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.075188    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 02 07:51:31.185382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.153564    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.185382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.154062    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:51:31.566651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.546373    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.620337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.619632    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 07:51:31.644812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.620848    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.709348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.686960    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.811880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.810735    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.828496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.823762    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:51:31.829259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.829234    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:51:31.832893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.832866    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:51:31.840866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.840831    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.868567 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.868537    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:51:31.882569 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.882532    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:51:31.886347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.886304    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:51:31.968518 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.968487    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:51:31.993153 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.993105    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.999840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:31.999810    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:51:32.015497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:32.014274    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:51:37.310101 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:37.310065    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:51:39.014598 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:39.014559    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:51:45.050334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:45.049423    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.606737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:45.606690    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.831334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:45.818391    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.895399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:45.889501    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.903840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:45.903421    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.055401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:46.052561    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.734197 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:51:47.734146    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:51:54.037068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.037031    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main" containerID="cri-o://257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e" gracePeriod=45
Nov 02 07:51:54.321362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.321328    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:51:54.394936 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.394906    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\" (UniqueName: \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.395149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.394981    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\" (UniqueName: \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.395149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.395023    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.395149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.395074    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.395358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.395274    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.395358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.395352    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\") pod \"47cea44e-69c0-4763-88d0-cf90e121410d\" (UID: \"47cea44e-69c0-4763-88d0-cf90e121410d\") "
Nov 02 07:51:54.396093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.396044    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:54.396231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.396098    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:54.396632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.396608    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:54.397913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.397881    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:54.399082 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.399055    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:54.399200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.399145    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7" pod "47cea44e-69c0-4763-88d0-cf90e121410d" (UID: "47cea44e-69c0-4763-88d0-cf90e121410d"). InnerVolumeSpecName "kube-api-access-vqpl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:54.497073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497030    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqpl7\" (UniqueName: \"kubernetes.io/projected/47cea44e-69c0-4763-88d0-cf90e121410d-kube-api-access-vqpl7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.497073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497067    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.497073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497080    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.497365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497118    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.497365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497130    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47cea44e-69c0-4763-88d0-cf90e121410d-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.497365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.497144    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cea44e-69c0-4763-88d0-cf90e121410d-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:51:54.576989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.576903    4515 generic.go:356] "Generic (PLEG): container finished" podID="47cea44e-69c0-4763-88d0-cf90e121410d" containerID="257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e" exitCode=0
Nov 02 07:51:54.576989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.576949    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"
Nov 02 07:51:54.577198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.576984    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" event={"ID":"47cea44e-69c0-4763-88d0-cf90e121410d","Type":"ContainerDied","Data":"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"}
Nov 02 07:51:54.577198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.577038    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh" event={"ID":"47cea44e-69c0-4763-88d0-cf90e121410d","Type":"ContainerDied","Data":"3ea94fee059c79e7287e247574c41d163b20a9f4ff979504efdcbc2cb602e34a"}
Nov 02 07:51:54.577198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.577061    4515 scope.go:122] "RemoveContainer" containerID="257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"
Nov 02 07:51:54.589656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.589633    4515 scope.go:122] "RemoveContainer" containerID="257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"
Nov 02 07:51:54.589957 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:51:54.589937    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e\": container with ID starting with 257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e not found: ID does not exist" containerID="257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"
Nov 02 07:51:54.590021 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.589967    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e"} err="failed to get container status \"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e\": rpc error: code = NotFound desc = could not find container \"257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e\": container with ID starting with 257bbed3a58e42c03985f9dd9d26d8e44889d405a1d6d189e15227c2f6e77a4e not found: ID does not exist"
Nov 02 07:51:54.630945 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.630915    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:51:54.643071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:54.643042    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-894bbb/praxis-route-894bbb-kserve-bb6966cc5-f5mgh"]
Nov 02 07:51:55.374331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:55.374285    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" path="/var/lib/kubelet/pods/47cea44e-69c0-4763-88d0-cf90e121410d/volumes"
Nov 02 07:51:59.030366 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.030260    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:51:59.031164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.031143    4515 cpu_manager.go:414] "containerMap: removing container" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main"
Nov 02 07:51:59.031164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.031165    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main"
Nov 02 07:51:59.031347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.031319    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cea44e-69c0-4763-88d0-cf90e121410d" containerName="main"
Nov 02 07:51:59.036152 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.036130    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.053129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.053105    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9ab4d2-openshift-default-dockercfg-xtcp6\""
Nov 02 07:51:59.072141 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.072107    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:51:59.156198 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156162    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddtb\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156232    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156444    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156508    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156577    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156604    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156694    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.156855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.156811    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258156    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258203    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258229    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258266    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258293    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258403    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258652    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258719    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258796 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258772    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.258948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258923    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddtb\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.259009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.258984    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.259115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.259097    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.259562 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.259535    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.259826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.259805    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.260990 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.260963    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.261158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.261143    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.282778 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.282723    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.282879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.282825    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddtb\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.350332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.350088    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:51:59.806046 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:51:59.806013    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdd76f7_d16a_4d74_bdb6_673f5495ba54.slice/crio-31a6887765a0150e05f48bd17ca674f07cf5ea235d51eaa50c430dbdddabe29e WatchSource:0}: Error finding container 31a6887765a0150e05f48bd17ca674f07cf5ea235d51eaa50c430dbdddabe29e: Status 404 returned error can't find the container with id 31a6887765a0150e05f48bd17ca674f07cf5ea235d51eaa50c430dbdddabe29e
Nov 02 07:51:59.806973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.806949    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:51:59.808609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.808572    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:59.808707 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.808675    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:59.808772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:51:59.808726    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:00.626355 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:00.626290    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerStarted","Data":"4614cfe98c4f9506db47ad84367cb91a0a61c76d814a7d5bbfc012c790a294d0"}
Nov 02 07:52:00.626816 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:00.626421    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerStarted","Data":"31a6887765a0150e05f48bd17ca674f07cf5ea235d51eaa50c430dbdddabe29e"}
Nov 02 07:52:00.687623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:00.687569    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podStartSLOduration=2.687554772 podStartE2EDuration="2.687554772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:00.685706627 +0000 UTC m=+1665.448143535" watchObservedRunningTime="2026-11-02 07:52:00.687554772 +0000 UTC m=+1665.449991677"
Nov 02 07:52:01.350959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:01.350918    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:01.357229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:01.357201    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:01.633864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:01.633769    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:01.634789 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:01.634764    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:05.116202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.116161    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:52:05.171604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.171562    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:52:05.171925 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.171911    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.221331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.221272    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.221950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.221922    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnr8\" (UniqueName: \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.238810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.238782    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9ab4d2\""
Nov 02 07:52:05.239170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.239129    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9ab4d2-dockercfg-x82xt\""
Nov 02 07:52:05.323527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.323276    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.323712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.323684    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnr8\" (UniqueName: \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.324026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.324001    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.347946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.347916    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnr8\" (UniqueName: \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\") pod \"payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.537493 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.537462    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:05.847085 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:52:05.847050    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e08179f_e24b_4eb6_b3f5_e0cd7ff3ccc4.slice/crio-0ae7494bcbee5d9b4d07cb9acef36d884c7b128f196474c7af15d1f7b27ea597 WatchSource:0}: Error finding container 0ae7494bcbee5d9b4d07cb9acef36d884c7b128f196474c7af15d1f7b27ea597: Status 404 returned error can't find the container with id 0ae7494bcbee5d9b4d07cb9acef36d884c7b128f196474c7af15d1f7b27ea597
Nov 02 07:52:05.847904 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:05.847874    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:52:06.704851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:06.704813    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" event={"ID":"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4","Type":"ContainerStarted","Data":"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"}
Nov 02 07:52:06.705232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:06.704862    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" event={"ID":"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4","Type":"ContainerStarted","Data":"0ae7494bcbee5d9b4d07cb9acef36d884c7b128f196474c7af15d1f7b27ea597"}
Nov 02 07:52:06.705232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:06.705038    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:06.751565 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:06.751512    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" podStartSLOduration=1.7514987130000002 podStartE2EDuration="1.751498713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:06.747450569 +0000 UTC m=+1671.509887491" watchObservedRunningTime="2026-11-02 07:52:06.751498713 +0000 UTC m=+1671.513935620"
Nov 02 07:52:08.543124 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:08.543091    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:52:16.513148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:16.513115    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:52:16.531893 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:16.531854    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:52:17.035381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:17.035343    4515 scope.go:122] "RemoveContainer" containerID="61648711f1e167b9a5f096c2f453bdfd02d8319994d20772ebd27dcb38716665"
Nov 02 07:52:17.715992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:17.715964    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:52:17.735415 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:17.735350    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:52:21.514397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:21.514329    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing" containerID="cri-o://8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60" gracePeriod=25
Nov 02 07:52:21.532290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:21.532233    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing" containerID="cri-o://744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968" gracePeriod=25
Nov 02 07:52:23.147388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.147353    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:52:23.153197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.153179    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.186994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.182466    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9ab4d2-serving-cert\""
Nov 02 07:52:23.186994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.186386    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:52:23.287445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.287399    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.287600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.287545    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkcq\" (UniqueName: \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.287600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.287582    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.287805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.287790    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.390364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.390322    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkcq\" (UniqueName: \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.391022 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.390994    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.391417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.391395    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.391631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.391616    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.393548 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:23.393409    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-serving-cert: secret "maas-api-e2e-praxis-b-9ab4d2-serving-cert" not found
Nov 02 07:52:23.393548 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:23.393480    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls podName:59dc47c6-8416-4378-920b-af25c69f334f nodeName:}" failed. No retries permitted until 2026-11-02 07:52:23.893459725 +0000 UTC m=+1688.655896612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls") pod "maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" (UID: "59dc47c6-8416-4378-920b-af25c69f334f") : secret "maas-api-e2e-praxis-b-9ab4d2-serving-cert" not found
Nov 02 07:52:23.394731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.394596    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.425410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.419253    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.425410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.424595    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkcq\" (UniqueName: \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.898418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.898379    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:23.900970 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:23.900944    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:24.067785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:24.067742    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:24.417186 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:52:24.417114    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dc47c6_8416_4378_920b_af25c69f334f.slice/crio-ae1cc52dd9b86196f52c42230dfbdfaeef081d5280fd0a871c28fc2a5206884c WatchSource:0}: Error finding container ae1cc52dd9b86196f52c42230dfbdfaeef081d5280fd0a871c28fc2a5206884c: Status 404 returned error can't find the container with id ae1cc52dd9b86196f52c42230dfbdfaeef081d5280fd0a871c28fc2a5206884c
Nov 02 07:52:24.417793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:24.417768    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:52:24.418594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:24.418575    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:52:24.854663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:24.854630    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" event={"ID":"59dc47c6-8416-4378-920b-af25c69f334f","Type":"ContainerStarted","Data":"ae1cc52dd9b86196f52c42230dfbdfaeef081d5280fd0a871c28fc2a5206884c"}
Nov 02 07:52:25.739896 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:25.739853    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:52:25.740388 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:25.739868    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:52:25.862378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:25.862343    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" event={"ID":"59dc47c6-8416-4378-920b-af25c69f334f","Type":"ContainerStarted","Data":"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"}
Nov 02 07:52:25.862584 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:25.862431    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:25.907987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:25.907936    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" podStartSLOduration=2.4829248760000002 podStartE2EDuration="2.907920592s" totalImagesPullingTime="424.995716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:25.905906217 +0000 UTC m=+1690.668343123" watchObservedRunningTime="2026-11-02 07:52:25.907920592 +0000 UTC m=+1690.670357498"
Nov 02 07:52:31.873684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:31.873649    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:52:34.845399 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:34.845347    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:34.898460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:34.898410    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:34.916119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:34.916075    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:34.971824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:34.971782    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.606342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.606265    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.607035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.606987    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.855530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.855477    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.881558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.862596    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:52:35.896344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.871689    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.906374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.901618    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.906374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.902072    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:52:35.919779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.919737    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:52:35.921376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.921126    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.927520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.927477    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:52:35.963772 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.961090    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 07:52:35.973666 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.973627    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:36.016362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:35.991578    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:52:36.207424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.112417    4515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 07:52:36.628230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.628172    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 02 07:52:36.656377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.631711    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:52:36.914185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.914080    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:36.996445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.996398    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 07:52:36.997433 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.997377    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 02 07:52:36.997929 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.997532    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": context deadline exceeded"
Nov 02 07:52:36.998429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:36.998323    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:52:37.078932 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.050875    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 07:52:37.089740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.089706    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 07:52:37.090482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.090460    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 07:52:37.093421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.091587    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 07:52:37.110922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.105975    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:52:37.175028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:37.174945    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 07:52:40.651335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:40.649813    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:52:40.651335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:40.650109    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" podUID="b80f0544-158e-4eff-b810-f80803306c73" containerName="maas-api" containerID="cri-o://f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e" gracePeriod=30
Nov 02 07:52:41.126321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.126094    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt_903e89cc-a05d-41cf-941e-539b23a06e4a/payload-processing/0.log"
Nov 02 07:52:41.159250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.158905    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:52:41.278499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.277076    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") pod \"b80f0544-158e-4eff-b810-f80803306c73\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") "
Nov 02 07:52:41.278499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.277195    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\" (UniqueName: \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\") pod \"b80f0544-158e-4eff-b810-f80803306c73\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") "
Nov 02 07:52:41.278499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.277359    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\") pod \"b80f0544-158e-4eff-b810-f80803306c73\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") "
Nov 02 07:52:41.278499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.277563    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\") pod \"b80f0544-158e-4eff-b810-f80803306c73\" (UID: \"b80f0544-158e-4eff-b810-f80803306c73\") "
Nov 02 07:52:41.280448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.280037    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca" pod "b80f0544-158e-4eff-b810-f80803306c73" (UID: "b80f0544-158e-4eff-b810-f80803306c73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:41.281427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.280742    4515 generic.go:356] "Generic (PLEG): container finished" podID="b80f0544-158e-4eff-b810-f80803306c73" containerID="f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e" exitCode=0
Nov 02 07:52:41.281427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.280919    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" event={"ID":"b80f0544-158e-4eff-b810-f80803306c73","Type":"ContainerDied","Data":"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"}
Nov 02 07:52:41.281427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.280957    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft" event={"ID":"b80f0544-158e-4eff-b810-f80803306c73","Type":"ContainerDied","Data":"f87b16d63d579c0d1c1475162198afb8e8e19dade6b91c9f2c6c225d6692fe20"}
Nov 02 07:52:41.281427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.280976    4515 scope.go:122] "RemoveContainer" containerID="f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"
Nov 02 07:52:41.281427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.281103    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"
Nov 02 07:52:41.282725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.282647    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls" pod "b80f0544-158e-4eff-b810-f80803306c73" (UID: "b80f0544-158e-4eff-b810-f80803306c73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:41.285503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.285476    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs" pod "b80f0544-158e-4eff-b810-f80803306c73" (UID: "b80f0544-158e-4eff-b810-f80803306c73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:41.285608 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.285552    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv" pod "b80f0544-158e-4eff-b810-f80803306c73" (UID: "b80f0544-158e-4eff-b810-f80803306c73"). InnerVolumeSpecName "kube-api-access-nfrfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:41.308703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.308678    4515 scope.go:122] "RemoveContainer" containerID="f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"
Nov 02 07:52:41.309112 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:41.309082    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e\": container with ID starting with f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e not found: ID does not exist" containerID="f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"
Nov 02 07:52:41.309200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.309124    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e"} err="failed to get container status \"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e\": rpc error: code = NotFound desc = could not find container \"f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e\": container with ID starting with f1276695181f275bcd530348c922e29af733730157a0cdd542fb521ca073d77e not found: ID does not exist"
Nov 02 07:52:41.357259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.357228    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwdh9_dad9dc47-790e-4c20-952e-c2984691c5d4/payload-processing/0.log"
Nov 02 07:52:41.383609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.379037    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.383609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.379062    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b80f0544-158e-4eff-b810-f80803306c73-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.383609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.379071    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrfv\" (UniqueName: \"kubernetes.io/projected/b80f0544-158e-4eff-b810-f80803306c73-kube-api-access-nfrfv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.383609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.379080    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80f0544-158e-4eff-b810-f80803306c73-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.584291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.584261    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwdh9_dad9dc47-790e-4c20-952e-c2984691c5d4/payload-processing/0.log"
Nov 02 07:52:41.656406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.656323    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:52:41.664420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.664387    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-894bbb-75dc6d8878-mndft"]
Nov 02 07:52:41.880679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.880649    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:52:41.933995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.933968    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:52:41.991073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991031    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\") pod \"9a004714-7dd3-40d4-95c6-769fc5e50e19\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") "
Nov 02 07:52:41.991243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\") pod \"da2351e5-affe-4450-8886-258cccc65413\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") "
Nov 02 07:52:41.991243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991185    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\" (UniqueName: \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\") pod \"9a004714-7dd3-40d4-95c6-769fc5e50e19\" (UID: \"9a004714-7dd3-40d4-95c6-769fc5e50e19\") "
Nov 02 07:52:41.991363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991276    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\" (UniqueName: \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\") pod \"da2351e5-affe-4450-8886-258cccc65413\" (UID: \"da2351e5-affe-4450-8886-258cccc65413\") "
Nov 02 07:52:41.991520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991489    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume" pod "9a004714-7dd3-40d4-95c6-769fc5e50e19" (UID: "9a004714-7dd3-40d4-95c6-769fc5e50e19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:41.992257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.991930    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume" pod "da2351e5-affe-4450-8886-258cccc65413" (UID: "da2351e5-affe-4450-8886-258cccc65413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:41.992257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.992044    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a004714-7dd3-40d4-95c6-769fc5e50e19-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.996473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.996244    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k" pod "da2351e5-affe-4450-8886-258cccc65413" (UID: "da2351e5-affe-4450-8886-258cccc65413"). InnerVolumeSpecName "kube-api-access-wmt6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:42.005347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:41.999770    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn" pod "9a004714-7dd3-40d4-95c6-769fc5e50e19" (UID: "9a004714-7dd3-40d4-95c6-769fc5e50e19"). InnerVolumeSpecName "kube-api-access-vqlpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:42.093667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.093552    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2351e5-affe-4450-8886-258cccc65413-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:42.093667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.093590    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqlpn\" (UniqueName: \"kubernetes.io/projected/9a004714-7dd3-40d4-95c6-769fc5e50e19-kube-api-access-vqlpn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:42.093667 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.093606    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmt6k\" (UniqueName: \"kubernetes.io/projected/da2351e5-affe-4450-8886-258cccc65413-kube-api-access-wmt6k\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:42.291091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.291059    4515 generic.go:356] "Generic (PLEG): container finished" podID="da2351e5-affe-4450-8886-258cccc65413" containerID="744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968" exitCode=0
Nov 02 07:52:42.291282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.291126    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"
Nov 02 07:52:42.291282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.291154    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" event={"ID":"da2351e5-affe-4450-8886-258cccc65413","Type":"ContainerDied","Data":"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"}
Nov 02 07:52:42.291282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.291216    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5" event={"ID":"da2351e5-affe-4450-8886-258cccc65413","Type":"ContainerDied","Data":"fc22f4ea7b348412c12e6a4295f4316f02beddde9e60835224156db12981d42a"}
Nov 02 07:52:42.291282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.291247    4515 scope.go:122] "RemoveContainer" containerID="744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"
Nov 02 07:52:42.294283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.294259    4515 generic.go:356] "Generic (PLEG): container finished" podID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerID="8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60" exitCode=0
Nov 02 07:52:42.294418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.294322    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" event={"ID":"9a004714-7dd3-40d4-95c6-769fc5e50e19","Type":"ContainerDied","Data":"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"}
Nov 02 07:52:42.294418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.294342    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"
Nov 02 07:52:42.294418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.294363    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b" event={"ID":"9a004714-7dd3-40d4-95c6-769fc5e50e19","Type":"ContainerDied","Data":"f6967ad85b88346c5a023839f52c2af8eac5dd834a9660c84943cba91899c22f"}
Nov 02 07:52:42.303157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.303126    4515 scope.go:122] "RemoveContainer" containerID="744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"
Nov 02 07:52:42.303533 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:42.303504    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968\": container with ID starting with 744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968 not found: ID does not exist" containerID="744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"
Nov 02 07:52:42.303641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.303545    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968"} err="failed to get container status \"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968\": rpc error: code = NotFound desc = could not find container \"744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968\": container with ID starting with 744103f62a240b31bf63d4bd33dd0aab99f3421cbb7ea7595cc7c1aa26552968 not found: ID does not exist"
Nov 02 07:52:42.303641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.303572    4515 scope.go:122] "RemoveContainer" containerID="8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"
Nov 02 07:52:42.314107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.314085    4515 scope.go:122] "RemoveContainer" containerID="8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"
Nov 02 07:52:42.314478 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:42.314457    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60\": container with ID starting with 8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60 not found: ID does not exist" containerID="8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"
Nov 02 07:52:42.314551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.314487    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60"} err="failed to get container status \"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60\": rpc error: code = NotFound desc = could not find container \"8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60\": container with ID starting with 8c406abc971a53593978881b9cab011692f5d259a6ba34ac3c1b683564647d60 not found: ID does not exist"
Nov 02 07:52:42.361100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.360989    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:52:42.373179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.373147    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-894bbb-54c8bb7567-922f5"]
Nov 02 07:52:42.410404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.410365    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:52:42.415961 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:42.415928    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-894bbb-797cb7cc4f-4hm9b"]
Nov 02 07:52:43.375620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:43.375587    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" path="/var/lib/kubelet/pods/9a004714-7dd3-40d4-95c6-769fc5e50e19/volumes"
Nov 02 07:52:43.376072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:43.376058    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80f0544-158e-4eff-b810-f80803306c73" path="/var/lib/kubelet/pods/b80f0544-158e-4eff-b810-f80803306c73/volumes"
Nov 02 07:52:43.376574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:43.376548    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2351e5-affe-4450-8886-258cccc65413" path="/var/lib/kubelet/pods/da2351e5-affe-4450-8886-258cccc65413/volumes"
Nov 02 07:52:44.227814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.227778    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:52:44.228587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228561    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing"
Nov 02 07:52:44.228587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228579    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228605    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b80f0544-158e-4eff-b810-f80803306c73" containerName="maas-api"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228610    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80f0544-158e-4eff-b810-f80803306c73" containerName="maas-api"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228625    4515 cpu_manager.go:414] "containerMap: removing container" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228631    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228722    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80f0544-158e-4eff-b810-f80803306c73" containerName="maas-api"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228731    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2351e5-affe-4450-8886-258cccc65413" containerName="payload-pre-processing"
Nov 02 07:52:44.228803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.228742    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a004714-7dd3-40d4-95c6-769fc5e50e19" containerName="payload-processing"
Nov 02 07:52:44.234059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.234027    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.258614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.257686    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:52:44.325855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.325807    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.325855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.325855    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.325988    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326049    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326074    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326197    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326428    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326463 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326456    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.326583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.326490    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qx4v\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.427696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427649    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427758    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427782    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427812    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qx4v\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427844    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427877    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427923    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.427987    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428025    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428248    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428280    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428418    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428494    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.428946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.428891    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.430468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.430446    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.430954 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.430934    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.452086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.452053    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.452232 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.452182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qx4v\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.562406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.562369    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:44.895283 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:52:44.895254    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fefbdc1_8411_4270_9814_120c80af3dd9.slice/crio-9fc386a6c0c22e7200d816abea11a4b355b6f23826c5c8210a29f6f9970c4733 WatchSource:0}: Error finding container 9fc386a6c0c22e7200d816abea11a4b355b6f23826c5c8210a29f6f9970c4733: Status 404 returned error can't find the container with id 9fc386a6c0c22e7200d816abea11a4b355b6f23826c5c8210a29f6f9970c4733
Nov 02 07:52:44.896849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.896822    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:52:44.897977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.897944    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.898044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.898010    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.898044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.898037    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.997642 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.997609    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:52:44.997944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:44.997898    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" containerID="cri-o://786cd89973a8eeb6e8d800ce75b7d44210fc52474b923c451c6ca3d4993aca06" gracePeriod=30
Nov 02 07:52:45.317849 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.317813    4515 generic.go:356] "Generic (PLEG): container finished" podID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerID="4614cfe98c4f9506db47ad84367cb91a0a61c76d814a7d5bbfc012c790a294d0" exitCode=0
Nov 02 07:52:45.318044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.317888    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerDied","Data":"4614cfe98c4f9506db47ad84367cb91a0a61c76d814a7d5bbfc012c790a294d0"}
Nov 02 07:52:45.318467 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.318445    4515 scope.go:122] "RemoveContainer" containerID="4614cfe98c4f9506db47ad84367cb91a0a61c76d814a7d5bbfc012c790a294d0"
Nov 02 07:52:45.319814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.319793    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" event={"ID":"1fefbdc1-8411-4270-9814-120c80af3dd9","Type":"ContainerStarted","Data":"2dfec31421f235b4ba1ed3adecae721ba9204e5983ed05a1cb2054630b67c45d"}
Nov 02 07:52:45.319922 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.319835    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" event={"ID":"1fefbdc1-8411-4270-9814-120c80af3dd9","Type":"ContainerStarted","Data":"9fc386a6c0c22e7200d816abea11a4b355b6f23826c5c8210a29f6f9970c4733"}
Nov 02 07:52:45.320280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.320240    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:45.320384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.320355    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:45.320426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.320409    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:45.414902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.414843    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podStartSLOduration=1.414824315 podStartE2EDuration="1.414824315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:45.411047716 +0000 UTC m=+1710.173484625" watchObservedRunningTime="2026-11-02 07:52:45.414824315 +0000 UTC m=+1710.177261222"
Nov 02 07:52:45.568844 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.568748    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:45.570730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.570412    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:45.570730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:45.570469    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:46.388682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:46.388642    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerStarted","Data":"64df4c7e23534a80cc21091e16cd87a777ea93947ac351bd441594033b26dd51"}
Nov 02 07:52:46.568414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:46.565736    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:46.568414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:46.565806    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:46.641393 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:46.640424    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:47.361334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.355595    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:47.361334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.355725    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 07:52:47.361334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.355771    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 07:52:47.414126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.414094    4515 patch_prober.go:29] interesting pod/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:52:47.414344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.414230    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:52:47.566334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.565353    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:47.566334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.565425    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:47.773375 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:52:47.773331    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:52:47.813690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:47.813660    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:52:48.356333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:48.355722    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 07:52:48.356333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:48.355788    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 07:52:48.565333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:48.564501    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:48.565333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:48.564577    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:49.353332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:49.352744    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 07:52:49.353332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:49.352812    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 07:52:49.656814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:49.656728    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:49.693103 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:49.691115    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:50.156112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.156062    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 07:52:50.202843 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.202806    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 07:52:50.241515 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.206488    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.329995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.270095    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-12df3c-openshift-default-dockercfg-vtmql\""
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349201    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2cr\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349355    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349459    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349546    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349571    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349597    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349672    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.349711    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.351291    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 07:52:50.359335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.351358    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.454837    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.454898    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.454985    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2cr\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455108    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455233    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455379    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455405    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.455429    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.490229    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.490608    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.514398 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.491246    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.600948 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.600877    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.656924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.636265    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2cr\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.656924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.652961    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.702147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.684083    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.742341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.736909    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:50.742341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.736990    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:50.771468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.771430    4515 generic.go:356] "Generic (PLEG): container finished" podID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerID="786cd89973a8eeb6e8d800ce75b7d44210fc52474b923c451c6ca3d4993aca06" exitCode=0
Nov 02 07:52:50.771775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.771752    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerDied","Data":"786cd89973a8eeb6e8d800ce75b7d44210fc52474b923c451c6ca3d4993aca06"}
Nov 02 07:52:50.771965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.771949    4515 scope.go:122] "RemoveContainer" containerID="db12f1c23ae5f698e6142b7d53b8b32bf9f9b786308144787fc7faddbdc76142"
Nov 02 07:52:50.778941 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.778861    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.781639 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.781610    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\") pod \"e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") " pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:50.985297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:50.985270    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:52:51.023044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.023002    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:51.199886 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.199853    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.200265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.200244    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.200504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.200437    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.200647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.200633    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.200811 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.200796    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.200949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.200934    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.201102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.201088    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.201213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.201199    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.201408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.201392    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\") pod \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\" (UID: \"ad88fc9a-9940-420d-8e60-4fe5c699ff42\") "
Nov 02 07:52:51.202754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.202731    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:52:51.203516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.203496    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:52:51.204746 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.204709    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:52:51.205295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.205261    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:51.205960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.205941    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:52:51.267390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.266043    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "kube-api-access-54922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:51.268775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.268747    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311650    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311685    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311700    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311716    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311729    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311744    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54922\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-kube-api-access-54922\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.311797 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.311759    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.337351 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.316044    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:51.346347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.340466    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy" pod "ad88fc9a-9940-420d-8e60-4fe5c699ff42" (UID: "ad88fc9a-9940-420d-8e60-4fe5c699ff42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:52:51.364552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.364068    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 07:52:51.364552 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.364139    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 07:52:51.413703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.413015    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.413703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.413041    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad88fc9a-9940-420d-8e60-4fe5c699ff42-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:52:51.579382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.571713    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 07:52:51.579382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.571784    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 07:52:51.779521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.779486    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"
Nov 02 07:52:51.779521 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.779504    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk" event={"ID":"ad88fc9a-9940-420d-8e60-4fe5c699ff42","Type":"ContainerDied","Data":"4387e76c2a26f29e92d075bdf9e3b36d8c820a5885bf04acd63113a4f699edaa"}
Nov 02 07:52:51.779776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.779569    4515 scope.go:122] "RemoveContainer" containerID="786cd89973a8eeb6e8d800ce75b7d44210fc52474b923c451c6ca3d4993aca06"
Nov 02 07:52:51.861420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.856677    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:52:51.861420 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.859643    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-894bbb-openshift-default-8d84c79f7-wqbgk"]
Nov 02 07:52:51.951886 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:52:51.951846    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6959f0f6_a227_47c5_a6e0_c445d7fc35e2.slice/crio-887cba62c46157b46a09849f84c9352438513bd58d32948bffb4a4b645448621 WatchSource:0}: Error finding container 887cba62c46157b46a09849f84c9352438513bd58d32948bffb4a4b645448621: Status 404 returned error can't find the container with id 887cba62c46157b46a09849f84c9352438513bd58d32948bffb4a4b645448621
Nov 02 07:52:51.952725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.952700    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 07:52:51.954726 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.954694    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:51.954823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.954757    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:51.954823 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:51.954785    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:52.364962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.364933    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:52.367419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.367396    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:52:52.571179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.571105    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:52.580364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.577820    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:52.580364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.578729    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:52:52.793324 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.793278    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerStarted","Data":"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"}
Nov 02 07:52:52.793522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.793387    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerStarted","Data":"887cba62c46157b46a09849f84c9352438513bd58d32948bffb4a4b645448621"}
Nov 02 07:52:52.841599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:52.841545    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podStartSLOduration=2.841531088 podStartE2EDuration="2.841531088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:52.839158087 +0000 UTC m=+1717.601594994" watchObservedRunningTime="2026-11-02 07:52:52.841531088 +0000 UTC m=+1717.603967993"
Nov 02 07:52:53.024346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:53.024062    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:53.035345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:53.033212    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:53.386417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:53.385200    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" path="/var/lib/kubelet/pods/ad88fc9a-9940-420d-8e60-4fe5c699ff42/volumes"
Nov 02 07:52:53.799609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:53.799579    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:53.800527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:53.800501    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:52:55.546503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.546473    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 07:52:55.547099 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547085    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.547154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547100    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.547154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547111    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.547154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547116    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.547257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547205    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.547257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.547214    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad88fc9a-9940-420d-8e60-4fe5c699ff42" containerName="istio-proxy"
Nov 02 07:52:55.551074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.551049    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.578522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.578494    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 07:52:55.593368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.593294    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9877ce\""
Nov 02 07:52:55.593529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.593387    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9877ce-dockercfg-78kdx\""
Nov 02 07:52:55.642618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.642582    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7b5\" (UniqueName: \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.642782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.642643    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.746964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.745216    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7b5\" (UniqueName: \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.746964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.745347    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.779336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.769667    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.833286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.824808    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7b5\" (UniqueName: \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\") pod \"payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:55.866093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:55.865611    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:56.232034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:56.232004    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 07:52:56.864354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:56.864290    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" event={"ID":"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80","Type":"ContainerStarted","Data":"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"}
Nov 02 07:52:56.864754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:56.864393    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:52:56.864754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:56.864416    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" event={"ID":"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80","Type":"ContainerStarted","Data":"51da1f02bfa683a8c5a6221865553eacb5a9fc0b0aac4712a1f15ef7830e8f5c"}
Nov 02 07:52:56.906678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:56.906594    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" podStartSLOduration=1.90658046 podStartE2EDuration="1.90658046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:56.905961323 +0000 UTC m=+1721.668398228" watchObservedRunningTime="2026-11-02 07:52:56.90658046 +0000 UTC m=+1721.669017365"
Nov 02 07:52:58.206207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.206163    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 07:52:58.210612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.210591    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.240106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.240078    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-12df3c\""
Nov 02 07:52:58.240106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.240090    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-12df3c-dockercfg-8dcpp\""
Nov 02 07:52:58.243916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.243892    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 07:52:58.341332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.339302    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 07:52:58.382336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.380528    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.382336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.380638    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw76k\" (UniqueName: \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.452340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.442619    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 07:52:58.452340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.443602    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.474166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.463077    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:52:58.482600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.481928    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.482600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.482055    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw76k\" (UniqueName: \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.482600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.482114    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q86f\" (UniqueName: \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.482600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.482156    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.485838 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.485764    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.511531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.511506    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw76k\" (UniqueName: \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\") pod \"payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.531472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.525578    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:58.583209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.583163    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q86f\" (UniqueName: \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.583422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.583230    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.584482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.584454    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.613492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.613300    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q86f\" (UniqueName: \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\") pod \"payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.791726 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.791694    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:58.874370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:58.871025    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 07:52:58.878371 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:52:58.878299    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a79ad55_c933_453f_8924_254ec0d0011e.slice/crio-42eb9b3e0d687152cb13eecafe97ff76e7df9e658b4e85a39610a5463cb1cbbc WatchSource:0}: Error finding container 42eb9b3e0d687152cb13eecafe97ff76e7df9e658b4e85a39610a5463cb1cbbc: Status 404 returned error can't find the container with id 42eb9b3e0d687152cb13eecafe97ff76e7df9e658b4e85a39610a5463cb1cbbc
Nov 02 07:52:59.162100 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.160140    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 07:52:59.845254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.845216    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:52:59.925369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.924236    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" event={"ID":"7a79ad55-c933-453f-8924-254ec0d0011e","Type":"ContainerStarted","Data":"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"}
Nov 02 07:52:59.925369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.924363    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" event={"ID":"7a79ad55-c933-453f-8924-254ec0d0011e","Type":"ContainerStarted","Data":"42eb9b3e0d687152cb13eecafe97ff76e7df9e658b4e85a39610a5463cb1cbbc"}
Nov 02 07:52:59.925369 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.924696    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:52:59.926624 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.926579    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" event={"ID":"c35ac2e0-4ee8-4752-9a1f-51229b40e979","Type":"ContainerStarted","Data":"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"}
Nov 02 07:52:59.926765 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.926637    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" event={"ID":"c35ac2e0-4ee8-4752-9a1f-51229b40e979","Type":"ContainerStarted","Data":"d18b9f36d7c8c63a986107c23d5fc943304ff69297e1f01cbd0b9e6c53f7c4ac"}
Nov 02 07:52:59.927563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.927047    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:52:59.968276 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:52:59.968228    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" podStartSLOduration=1.968215994 podStartE2EDuration="1.968215994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:59.965947333 +0000 UTC m=+1724.728384238" watchObservedRunningTime="2026-11-02 07:52:59.968215994 +0000 UTC m=+1724.730652901"
Nov 02 07:53:00.009858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:00.009808    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" podStartSLOduration=2.00979036 podStartE2EDuration="2.00979036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:00.007114394 +0000 UTC m=+1724.769551300" watchObservedRunningTime="2026-11-02 07:53:00.00979036 +0000 UTC m=+1724.772227270"
Nov 02 07:53:01.862732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:01.862692    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:53:01.863236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:01.862948    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerName="maas-api" containerID="cri-o://c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a" gracePeriod=30
Nov 02 07:53:02.314873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.314846    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:53:02.456239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.456197    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\") pod \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") "
Nov 02 07:53:02.456469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.456261    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\") pod \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") "
Nov 02 07:53:02.456469 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.456392    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\" (UniqueName: \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\") pod \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") "
Nov 02 07:53:02.456647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.456621    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\") pod \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\" (UID: \"c5d7e05b-e821-495c-9ee2-a83dbdc349eb\") "
Nov 02 07:53:02.457063 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.457026    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca" pod "c5d7e05b-e821-495c-9ee2-a83dbdc349eb" (UID: "c5d7e05b-e821-495c-9ee2-a83dbdc349eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:02.457340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.457304    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:02.458438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.458415    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls" pod "c5d7e05b-e821-495c-9ee2-a83dbdc349eb" (UID: "c5d7e05b-e821-495c-9ee2-a83dbdc349eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:02.458950 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.458932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs" pod "c5d7e05b-e821-495c-9ee2-a83dbdc349eb" (UID: "c5d7e05b-e821-495c-9ee2-a83dbdc349eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:02.458994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.458960    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw" pod "c5d7e05b-e821-495c-9ee2-a83dbdc349eb" (UID: "c5d7e05b-e821-495c-9ee2-a83dbdc349eb"). InnerVolumeSpecName "kube-api-access-fz4jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:02.564653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.560543    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:02.564653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.560583    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:02.564653 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.560598    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4jw\" (UniqueName: \"kubernetes.io/projected/c5d7e05b-e821-495c-9ee2-a83dbdc349eb-kube-api-access-fz4jw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:02.886784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.886749    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" containerID="cri-o://0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91" gracePeriod=45
Nov 02 07:53:02.964781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.964735    4515 generic.go:356] "Generic (PLEG): container finished" podID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerID="c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a" exitCode=0
Nov 02 07:53:02.972159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.965222    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" event={"ID":"c5d7e05b-e821-495c-9ee2-a83dbdc349eb","Type":"ContainerDied","Data":"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"}
Nov 02 07:53:02.996333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.971738    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"
Nov 02 07:53:02.996333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.991894    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" event={"ID":"c5d7e05b-e821-495c-9ee2-a83dbdc349eb","Type":"ContainerDied","Data":"fefc73122d40fa0bc16684b68a5259ceecdb79cf5a1717166080c87332e08c5b"}
Nov 02 07:53:02.996333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:02.991966    4515 scope.go:122] "RemoveContainer" containerID="c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"
Nov 02 07:53:03.033937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.033913    4515 scope.go:122] "RemoveContainer" containerID="c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"
Nov 02 07:53:03.034252 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:03.034233    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a\": container with ID starting with c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a not found: ID does not exist" containerID="c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"
Nov 02 07:53:03.034341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.034262    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a"} err="failed to get container status \"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a\": rpc error: code = NotFound desc = could not find container \"c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a\": container with ID starting with c0650aaeaa49178c877a2cbf6e34b91d5cdd091121b2704de0a68ade3cd3bd3a not found: ID does not exist"
Nov 02 07:53:03.067075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.067040    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:53:03.079066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.079036    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l"]
Nov 02 07:53:03.204447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.204424    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:53:03.293522 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293491    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293608    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293650    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293716    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293786    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293827    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\" (UniqueName: \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\") pod \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\" (UID: \"b2a67473-c35a-43a7-8e77-bbb66098bbb1\") "
Nov 02 07:53:03.293898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.293891    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:03.294218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.294192    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:03.294443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.294420    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:03.294643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.294626    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.294724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.294674    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.294724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.294692    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.296118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.296084    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:03.296792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.296762    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "kube-api-access-z7fpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:03.296926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.296813    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs" pod "b2a67473-c35a-43a7-8e77-bbb66098bbb1" (UID: "b2a67473-c35a-43a7-8e77-bbb66098bbb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:03.374703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.374669    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" path="/var/lib/kubelet/pods/c5d7e05b-e821-495c-9ee2-a83dbdc349eb/volumes"
Nov 02 07:53:03.395779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.395752    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a67473-c35a-43a7-8e77-bbb66098bbb1-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.395779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.395780    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7fpn\" (UniqueName: \"kubernetes.io/projected/b2a67473-c35a-43a7-8e77-bbb66098bbb1-kube-api-access-z7fpn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.395951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.395790    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a67473-c35a-43a7-8e77-bbb66098bbb1-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.464447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.464360    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" podUID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" containerName="payload-processing" containerID="cri-o://1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc" gracePeriod=25
Nov 02 07:53:03.714384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.714359    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:53:03.903321 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.903290    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\") pod \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") "
Nov 02 07:53:03.903675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.903445    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\" (UniqueName: \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\") pod \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\" (UID: \"fa2f9d27-4b21-42fe-9219-e996b1d1097f\") "
Nov 02 07:53:03.903718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.903675    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume" pod "fa2f9d27-4b21-42fe-9219-e996b1d1097f" (UID: "fa2f9d27-4b21-42fe-9219-e996b1d1097f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:03.903873 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.903858    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f9d27-4b21-42fe-9219-e996b1d1097f-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.905513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.905488    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj" pod "fa2f9d27-4b21-42fe-9219-e996b1d1097f" (UID: "fa2f9d27-4b21-42fe-9219-e996b1d1097f"). InnerVolumeSpecName "kube-api-access-2q7jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:03.973920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.973889    4515 generic.go:356] "Generic (PLEG): container finished" podID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerID="0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91" exitCode=0
Nov 02 07:53:03.974114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.973933    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"
Nov 02 07:53:03.974114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.973973    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" event={"ID":"b2a67473-c35a-43a7-8e77-bbb66098bbb1","Type":"ContainerDied","Data":"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"}
Nov 02 07:53:03.974114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.974010    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" event={"ID":"b2a67473-c35a-43a7-8e77-bbb66098bbb1","Type":"ContainerDied","Data":"aebaa82fb41a4a8c16165cd09310eca7add981930c55c0ac5399f4c510306c18"}
Nov 02 07:53:03.974114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.974031    4515 scope.go:122] "RemoveContainer" containerID="0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"
Nov 02 07:53:03.975484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.975457    4515 generic.go:356] "Generic (PLEG): container finished" podID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" containerID="1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc" exitCode=0
Nov 02 07:53:03.975629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.975513    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"
Nov 02 07:53:03.975629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.975541    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" event={"ID":"fa2f9d27-4b21-42fe-9219-e996b1d1097f","Type":"ContainerDied","Data":"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"}
Nov 02 07:53:03.975629 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.975594    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5" event={"ID":"fa2f9d27-4b21-42fe-9219-e996b1d1097f","Type":"ContainerDied","Data":"17c5a2d75abe98e2e500a903db0a3908f354eb65117573a767ac80c752f2fe0a"}
Nov 02 07:53:03.984673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.984653    4515 scope.go:122] "RemoveContainer" containerID="0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"
Nov 02 07:53:03.984945 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:03.984928    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91\": container with ID starting with 0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91 not found: ID does not exist" containerID="0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"
Nov 02 07:53:03.985018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.984952    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91"} err="failed to get container status \"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91\": rpc error: code = NotFound desc = could not find container \"0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91\": container with ID starting with 0f18164fa83bbc38622ce4a342aa1a484e786808da8e5ee4b08aef8a05192d91 not found: ID does not exist"
Nov 02 07:53:03.985018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.984968    4515 scope.go:122] "RemoveContainer" containerID="1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"
Nov 02 07:53:03.993968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.993949    4515 scope.go:122] "RemoveContainer" containerID="1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"
Nov 02 07:53:03.994246 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:03.994225    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc\": container with ID starting with 1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc not found: ID does not exist" containerID="1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"
Nov 02 07:53:03.994320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:03.994257    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc"} err="failed to get container status \"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc\": rpc error: code = NotFound desc = could not find container \"1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc\": container with ID starting with 1c7e29433cb6ebddef591873b4e6df7e57dfe932c6d427027d5e5216a1e406bc not found: ID does not exist"
Nov 02 07:53:04.005234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.005207    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q7jj\" (UniqueName: \"kubernetes.io/projected/fa2f9d27-4b21-42fe-9219-e996b1d1097f-kube-api-access-2q7jj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:04.027491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.027462    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:53:04.040554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.040527    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt"]
Nov 02 07:53:04.066287 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.066259    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:53:04.070668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.070645    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-11b2e0-57f6455547-2zhr5"]
Nov 02 07:53:04.110215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:04.110175    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-16e21e/ipp-route-16e21e-kserve-5dfd49856f-4fqqt" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": context deadline exceeded"
Nov 02 07:53:05.298149 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:05.298103    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3624d8-67f66896b4-5bw6l" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.170:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:53:05.373508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:05.373470    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" path="/var/lib/kubelet/pods/b2a67473-c35a-43a7-8e77-bbb66098bbb1/volumes"
Nov 02 07:53:05.373986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:05.373963    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" path="/var/lib/kubelet/pods/fa2f9d27-4b21-42fe-9219-e996b1d1097f/volumes"
Nov 02 07:53:07.873981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:07.873953    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 07:53:10.937511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:10.937482    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 07:53:10.937887 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:10.937549    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 07:53:12.529444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:12.529404    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:53:12.530115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:12.529675    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" podUID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" containerName="maas-api" containerID="cri-o://cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71" gracePeriod=30
Nov 02 07:53:12.971229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:12.970842    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:53:13.091175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.091125    4515 generic.go:356] "Generic (PLEG): container finished" podID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" containerID="cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71" exitCode=0
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.091796    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.091818    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" event={"ID":"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f","Type":"ContainerDied","Data":"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"}
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.098816    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l" event={"ID":"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f","Type":"ContainerDied","Data":"bcb20c09102ce8590bae3e2006b8d09cb7ca590ec13075902ec4e5d8645fb59e"}
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.098952    4515 scope.go:122] "RemoveContainer" containerID="cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.092784    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\" (UniqueName: \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\") pod \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") "
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.101406    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") pod \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") "
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.101504    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\") pod \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") "
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.101603    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\") pod \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\" (UID: \"c3f3f36e-4408-4a22-a511-1dfcf9a50b9f\") "
Nov 02 07:53:13.108347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.102726    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca" pod "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" (UID: "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:13.135017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.134963    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq" pod "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" (UID: "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f"). InnerVolumeSpecName "kube-api-access-ctxsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:13.138524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.137487    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls" pod "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" (UID: "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:13.138959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.138932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs" pod "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" (UID: "c3f3f36e-4408-4a22-a511-1dfcf9a50b9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:13.151223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.151119    4515 scope.go:122] "RemoveContainer" containerID="cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"
Nov 02 07:53:13.151614 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:13.151516    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71\": container with ID starting with cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71 not found: ID does not exist" containerID="cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"
Nov 02 07:53:13.151614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.151557    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71"} err="failed to get container status \"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71\": rpc error: code = NotFound desc = could not find container \"cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71\": container with ID starting with cea3f4a3a9fa49195d033cb8c4ed771c788417654681f68ad3e6daad49cd9a71 not found: ID does not exist"
Nov 02 07:53:13.203286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.203250    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxsq\" (UniqueName: \"kubernetes.io/projected/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-kube-api-access-ctxsq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:13.203286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.203283    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:13.203286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.203293    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:13.203553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.203302    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:13.341528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.341494    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 07:53:13.342118 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342104    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main"
Nov 02 07:53:13.342163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342120    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main"
Nov 02 07:53:13.342163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342134    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" containerName="maas-api"
Nov 02 07:53:13.342163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342140    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" containerName="maas-api"
Nov 02 07:53:13.342163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342149    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" containerName="payload-processing"
Nov 02 07:53:13.342163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342154    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" containerName="payload-processing"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342180    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerName="maas-api"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342185    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerName="maas-api"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342290    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d7e05b-e821-495c-9ee2-a83dbdc349eb" containerName="maas-api"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342303    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a67473-c35a-43a7-8e77-bbb66098bbb1" containerName="main"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342320    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" containerName="maas-api"
Nov 02 07:53:13.342361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.342325    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2f9d27-4b21-42fe-9219-e996b1d1097f" containerName="payload-processing"
Nov 02 07:53:13.347034 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.347015    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.364862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.364837    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9877ce-serving-cert\""
Nov 02 07:53:13.365488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.365463    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 07:53:13.443411 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.443300    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:53:13.448960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.448934    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-11b2e0-6dcf458df9-glb6l"]
Nov 02 07:53:13.507532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.507444    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.507532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.507665    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrh9w\" (UniqueName: \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.507532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.507715    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.508057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.507864    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.620775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.615525    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.620775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.615747    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh9w\" (UniqueName: \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.620775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.615793    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.620775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.615866    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.624267 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.622338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.661780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.661706    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh9w\" (UniqueName: \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.677703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.677671    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.680719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.677700    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:13.961039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:13.960998    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:14.300479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.300441    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 07:53:14.312516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.312486    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:14.318197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.318172    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.342647 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.342615    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:14.457525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457483    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.457734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457545    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.457734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457580    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm67s\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.457734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457691    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.457907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457764    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.457907 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.457870    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.458031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.458009    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.458096 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.458080    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.458139 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.458112    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.568503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.566673    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.632907    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633034    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633096    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633134    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm67s\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633197    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633273    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633428    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.633547    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.634486    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.646460 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.646387    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.647650 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.647601    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.648130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.648108    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.648676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.648586    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.653272 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.653248    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.653485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.653464    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.669370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.669343    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm67s\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.671918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.671892    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:14.937486 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:14.937390    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:15.114208 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.114175    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" event={"ID":"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d","Type":"ContainerStarted","Data":"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"}
Nov 02 07:53:15.114485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.114235    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" event={"ID":"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d","Type":"ContainerStarted","Data":"a6834d8aa215ffaa4930298a8a8fe1ad28605d97336d4839c07d7a6a0aac6b10"}
Nov 02 07:53:15.114485 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.114398    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:15.157470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.157416    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" podStartSLOduration=1.752771914 podStartE2EDuration="2.157402542s" totalImagesPullingTime="404.630628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:15.156496615 +0000 UTC m=+1739.918933522" watchObservedRunningTime="2026-11-02 07:53:15.157402542 +0000 UTC m=+1739.919839449"
Nov 02 07:53:15.322368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.319198    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:15.326996 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:15.326950    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1d9f7b_82af_4188_8148_723abaa4abf6.slice/crio-c94c9017bfd54d4b17d539528fc791937a1ff810af16b2bec5b0650411eb0858 WatchSource:0}: Error finding container c94c9017bfd54d4b17d539528fc791937a1ff810af16b2bec5b0650411eb0858: Status 404 returned error can't find the container with id c94c9017bfd54d4b17d539528fc791937a1ff810af16b2bec5b0650411eb0858
Nov 02 07:53:15.339996 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.339957    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.340133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.340034    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.340133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.340068    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.375472 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:15.375438    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f3f36e-4408-4a22-a511-1dfcf9a50b9f" path="/var/lib/kubelet/pods/c3f3f36e-4408-4a22-a511-1dfcf9a50b9f/volumes"
Nov 02 07:53:16.145952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.145919    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:53:16.147199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.147169    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" event={"ID":"8e1d9f7b-82af-4188-8148-723abaa4abf6","Type":"ContainerStarted","Data":"ae585f84030ca06002152d1849f7a9356a65eb9c6c50da7772af1d024d6f0d23"}
Nov 02 07:53:16.147341 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.147221    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" event={"ID":"8e1d9f7b-82af-4188-8148-723abaa4abf6","Type":"ContainerStarted","Data":"c94c9017bfd54d4b17d539528fc791937a1ff810af16b2bec5b0650411eb0858"}
Nov 02 07:53:16.247333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.246522    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podStartSLOduration=2.246501056 podStartE2EDuration="2.246501056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:16.234508571 +0000 UTC m=+1740.996945477" watchObservedRunningTime="2026-11-02 07:53:16.246501056 +0000 UTC m=+1741.008937962"
Nov 02 07:53:16.940331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.939786    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:16.948364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.944243    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 07:53:16.948364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:16.944299    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 07:53:17.516090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.516057    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 07:53:17.547620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.547585    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.553810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.553781    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 07:53:17.563417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.563390    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-12df3c-serving-cert\""
Nov 02 07:53:17.616017 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.615983    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zlp\" (UniqueName: \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.616229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.616140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.631334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.616289    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.631334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.616389    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.722334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.721278    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.722334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.721366    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.722334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.721509    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zlp\" (UniqueName: \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.722334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.721618    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.743484 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.743447    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.743997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.743941    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.746582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.746557    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.748360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.748243    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zlp\" (UniqueName: \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\") pod \"maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.749703 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:17.749677    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:53:17.868430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.868230    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:17.938523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.938021    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 07:53:17.938523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:17.938081    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 07:53:18.247527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:18.247492    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 07:53:18.248134 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:18.248059    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7be5b5_9c26_43fd_9026_06dcc11ce70f.slice/crio-45452f64adf631b3b5fb41a3e18a4bcc81f647f0a9b476940ba16bba02dac0b4 WatchSource:0}: Error finding container 45452f64adf631b3b5fb41a3e18a4bcc81f647f0a9b476940ba16bba02dac0b4: Status 404 returned error can't find the container with id 45452f64adf631b3b5fb41a3e18a4bcc81f647f0a9b476940ba16bba02dac0b4
Nov 02 07:53:18.941333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:18.940721    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 07:53:18.941333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:18.940787    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 07:53:19.180384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.180343    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" event={"ID":"ec7be5b5-9c26-43fd-9026-06dcc11ce70f","Type":"ContainerStarted","Data":"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"}
Nov 02 07:53:19.180589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.180410    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" event={"ID":"ec7be5b5-9c26-43fd-9026-06dcc11ce70f","Type":"ContainerStarted","Data":"45452f64adf631b3b5fb41a3e18a4bcc81f647f0a9b476940ba16bba02dac0b4"}
Nov 02 07:53:19.180589 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.180453    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:19.239380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.238125    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" podStartSLOduration=1.896247365 podStartE2EDuration="2.238104107s" totalImagesPullingTime="341.856742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:19.22878739 +0000 UTC m=+1743.991224327" watchObservedRunningTime="2026-11-02 07:53:19.238104107 +0000 UTC m=+1744.000541013"
Nov 02 07:53:19.943374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.942344    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 07:53:19.943374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:19.942411    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 07:53:20.871011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:20.870971    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:53:20.871918 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:20.871882    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy" containerID="cri-o://699e89403593006c4e0dade9afd6c4dab494d75ad9b2fe5c66846eb40719705a" gracePeriod=30
Nov 02 07:53:20.982986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:20.978458    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 07:53:20.982986 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:20.978524    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 07:53:21.146779 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.146689    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" podUID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" containerName="payload-processing" containerID="cri-o://0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2" gracePeriod=25
Nov 02 07:53:21.153786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.153759    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 07:53:21.490784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.490755    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:53:21.602444 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.602417    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\" (UniqueName: \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\") pod \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") "
Nov 02 07:53:21.602635 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.602618    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\") pod \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\" (UID: \"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4\") "
Nov 02 07:53:21.603038 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.603008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume" pod "1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" (UID: "1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:21.603205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.603128    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:21.604694 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.604660    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8" pod "1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" (UID: "1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4"). InnerVolumeSpecName "kube-api-access-btnr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:21.704890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:21.704802    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnr8\" (UniqueName: \"kubernetes.io/projected/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4-kube-api-access-btnr8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:22.027050 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.026133    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:22.027668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.027643    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:22.028742 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.028720    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:22.201490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.201451    4515 generic.go:356] "Generic (PLEG): container finished" podID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" containerID="0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2" exitCode=0
Nov 02 07:53:22.201659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.201515    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"
Nov 02 07:53:22.201659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.201535    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" event={"ID":"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4","Type":"ContainerDied","Data":"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"}
Nov 02 07:53:22.201659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.201629    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk" event={"ID":"1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4","Type":"ContainerDied","Data":"0ae7494bcbee5d9b4d07cb9acef36d884c7b128f196474c7af15d1f7b27ea597"}
Nov 02 07:53:22.201659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.201658    4515 scope.go:122] "RemoveContainer" containerID="0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"
Nov 02 07:53:22.213051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.213031    4515 scope.go:122] "RemoveContainer" containerID="0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"
Nov 02 07:53:22.213367 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:22.213298    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2\": container with ID starting with 0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2 not found: ID does not exist" containerID="0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"
Nov 02 07:53:22.213405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.213383    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2"} err="failed to get container status \"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2\": rpc error: code = NotFound desc = could not find container \"0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2\": container with ID starting with 0ecb6b17d67296376d9148b77142869ea717121b9ef864fede64c4b1e2825bd2 not found: ID does not exist"
Nov 02 07:53:22.311397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.309197    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:53:22.320332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:22.319056    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9ab4d2-5fc987db45-xp2gk"]
Nov 02 07:53:23.373847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:23.373812    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" path="/var/lib/kubelet/pods/1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4/volumes"
Nov 02 07:53:25.193965 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:25.193938    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 07:53:26.319951 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.319924    4515 generic.go:356] "Generic (PLEG): container finished" podID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerID="699e89403593006c4e0dade9afd6c4dab494d75ad9b2fe5c66846eb40719705a" exitCode=0
Nov 02 07:53:26.320549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.320013    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" event={"ID":"5429a7df-22b6-456a-8ab7-383a0ac06aa4","Type":"ContainerDied","Data":"699e89403593006c4e0dade9afd6c4dab494d75ad9b2fe5c66846eb40719705a"}
Nov 02 07:53:26.448205 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.448176    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:53:26.482619 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482581    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482632    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482670    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482772    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482832    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482865    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.482976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.482967    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.483127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483034    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.483127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483105    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.483235 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483135    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:26.483298 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483236    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\") pod \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\" (UID: \"5429a7df-22b6-456a-8ab7-383a0ac06aa4\") "
Nov 02 07:53:26.483429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483396    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.483429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483416    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.483678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.483646    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.484106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484082    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.484145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484115    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.484145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484130    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.484215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484143    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.484215 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484159    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.484939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.484912    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.485094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.485075    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:26.485391 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.485370    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "kube-api-access-m9rqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:26.485888 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.485872    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo" pod "5429a7df-22b6-456a-8ab7-383a0ac06aa4" (UID: "5429a7df-22b6-456a-8ab7-383a0ac06aa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:26.586165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.586027    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-kube-api-access-m9rqq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.586165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.586060    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.586165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.586073    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.586165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:26.586086    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5429a7df-22b6-456a-8ab7-383a0ac06aa4-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:27.329854 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:27.329825    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"
Nov 02 07:53:27.330363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:27.329850    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk" event={"ID":"5429a7df-22b6-456a-8ab7-383a0ac06aa4","Type":"ContainerDied","Data":"b06c4fd3278a9da989bff0224ee162fefa9ccca508eba25de0f49e96aabf6c7c"}
Nov 02 07:53:27.330363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:27.329922    4515 scope.go:122] "RemoveContainer" containerID="699e89403593006c4e0dade9afd6c4dab494d75ad9b2fe5c66846eb40719705a"
Nov 02 07:53:27.396783 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:27.396745    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:53:27.409295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:27.409266    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-99lqk"]
Nov 02 07:53:29.351335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.351288    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:29.351955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.351940    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" containerName="payload-processing"
Nov 02 07:53:29.351999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.351955    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" containerName="payload-processing"
Nov 02 07:53:29.351999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.351978    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy"
Nov 02 07:53:29.351999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.351983    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy"
Nov 02 07:53:29.352090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.352083    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e08179f-e24b-4eb6-b3f5-e0cd7ff3ccc4" containerName="payload-processing"
Nov 02 07:53:29.352129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.352094    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" containerName="istio-proxy"
Nov 02 07:53:29.356847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.356830    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.377197 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.377170    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5429a7df-22b6-456a-8ab7-383a0ac06aa4" path="/var/lib/kubelet/pods/5429a7df-22b6-456a-8ab7-383a0ac06aa4/volumes"
Nov 02 07:53:29.393655 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.393627    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:29.427041 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427008    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvwh\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427067    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427190    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427222    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427349    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427375    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427581    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.427669 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.427632    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529255    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529322    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529394    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529446    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529508    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529571    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvwh\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529658 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529626    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529738    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529820 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529718    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529825    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.529937 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.529859    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.530207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.530172    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.530274 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.530188    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.531864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.531842    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.532162 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.532145    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.553413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.553384    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.553510 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.553447    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvwh\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.675835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.675753    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:29.997535 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:29.997498    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b86f17_ddc4_4e6b_9ed1_dd232c724260.slice/crio-5593d79573e504e6f10a1d1dc8b0d8a48eb109e6d695db66db3fbad9ed33a031 WatchSource:0}: Error finding container 5593d79573e504e6f10a1d1dc8b0d8a48eb109e6d695db66db3fbad9ed33a031: Status 404 returned error can't find the container with id 5593d79573e504e6f10a1d1dc8b0d8a48eb109e6d695db66db3fbad9ed33a031
Nov 02 07:53:29.998200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:29.998169    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:30.000448 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.000410    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:30.000531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.000483    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:30.000531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.000515    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:30.381993 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.381956    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" event={"ID":"53b86f17-ddc4-4e6b-9ed1-dd232c724260","Type":"ContainerStarted","Data":"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"}
Nov 02 07:53:30.382497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.382021    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" event={"ID":"53b86f17-ddc4-4e6b-9ed1-dd232c724260","Type":"ContainerStarted","Data":"5593d79573e504e6f10a1d1dc8b0d8a48eb109e6d695db66db3fbad9ed33a031"}
Nov 02 07:53:30.428510 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.428454    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podStartSLOduration=1.428439234 podStartE2EDuration="1.428439234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:30.427648801 +0000 UTC m=+1755.190085714" watchObservedRunningTime="2026-11-02 07:53:30.428439234 +0000 UTC m=+1755.190876139"
Nov 02 07:53:30.676853 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:30.676768    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:31.488573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:31.488541    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:53:31.489039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:31.488761    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" podUID="59dc47c6-8416-4378-920b-af25c69f334f" containerName="maas-api" containerID="cri-o://0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52" gracePeriod=30
Nov 02 07:53:31.678204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:31.678171    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:31.678408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:31.678225    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:32.076339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.075513    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:53:32.191952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.191912    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") pod \"59dc47c6-8416-4378-920b-af25c69f334f\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") "
Nov 02 07:53:32.210572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.210492    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\" (UniqueName: \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\") pod \"59dc47c6-8416-4378-920b-af25c69f334f\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") "
Nov 02 07:53:32.210572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.210558    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\") pod \"59dc47c6-8416-4378-920b-af25c69f334f\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") "
Nov 02 07:53:32.210808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.210706    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\") pod \"59dc47c6-8416-4378-920b-af25c69f334f\" (UID: \"59dc47c6-8416-4378-920b-af25c69f334f\") "
Nov 02 07:53:32.211899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.209219    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls" pod "59dc47c6-8416-4378-920b-af25c69f334f" (UID: "59dc47c6-8416-4378-920b-af25c69f334f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:32.212387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.212360    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca" pod "59dc47c6-8416-4378-920b-af25c69f334f" (UID: "59dc47c6-8416-4378-920b-af25c69f334f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:32.215601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.215575    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq" pod "59dc47c6-8416-4378-920b-af25c69f334f" (UID: "59dc47c6-8416-4378-920b-af25c69f334f"). InnerVolumeSpecName "kube-api-access-9bkcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:32.217135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.217108    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs" pod "59dc47c6-8416-4378-920b-af25c69f334f" (UID: "59dc47c6-8416-4378-920b-af25c69f334f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:32.312207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.312173    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bkcq\" (UniqueName: \"kubernetes.io/projected/59dc47c6-8416-4378-920b-af25c69f334f-kube-api-access-9bkcq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.312207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.312206    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.312468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.312222    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59dc47c6-8416-4378-920b-af25c69f334f-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.312468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.312233    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59dc47c6-8416-4378-920b-af25c69f334f-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.398170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.398135    4515 generic.go:356] "Generic (PLEG): container finished" podID="59dc47c6-8416-4378-920b-af25c69f334f" containerID="0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52" exitCode=0
Nov 02 07:53:32.398367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.398184    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"
Nov 02 07:53:32.398367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.398221    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" event={"ID":"59dc47c6-8416-4378-920b-af25c69f334f","Type":"ContainerDied","Data":"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"}
Nov 02 07:53:32.398367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.398282    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" event={"ID":"59dc47c6-8416-4378-920b-af25c69f334f","Type":"ContainerDied","Data":"ae1cc52dd9b86196f52c42230dfbdfaeef081d5280fd0a871c28fc2a5206884c"}
Nov 02 07:53:32.398367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.398334    4515 scope.go:122] "RemoveContainer" containerID="0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"
Nov 02 07:53:32.408938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.408919    4515 scope.go:122] "RemoveContainer" containerID="0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"
Nov 02 07:53:32.409175 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:32.409152    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52\": container with ID starting with 0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52 not found: ID does not exist" containerID="0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"
Nov 02 07:53:32.409243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.409186    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52"} err="failed to get container status \"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52\": rpc error: code = NotFound desc = could not find container \"0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52\": container with ID starting with 0f92f8b6c0bd957a7c0cf7300418c76672216390a09d8459d35f5d7ebaef4d52 not found: ID does not exist"
Nov 02 07:53:32.456209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.456170    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:53:32.459776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.459749    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm"]
Nov 02 07:53:32.676498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.676467    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:32.676998 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:32.676520    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:33.373957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:33.373926    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dc47c6-8416-4378-920b-af25c69f334f" path="/var/lib/kubelet/pods/59dc47c6-8416-4378-920b-af25c69f334f/volumes"
Nov 02 07:53:33.677018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:33.676926    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:33.677018 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:33.676988    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:34.676628 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:34.676591    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:34.676815 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:34.676654    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:34.910035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:34.909986    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9ab4d2-9bcf686-l9phm" podUID="59dc47c6-8416-4378-920b-af25c69f334f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.173:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:53:35.676520 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:35.676486    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:35.676705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:35.676556    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:36.676579 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:36.676546    4515 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 07:53:36.677000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:36.676602    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 07:53:37.681253 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:37.681218    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:37.681676 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:37.681503    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:37.682230 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:37.682208    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:40.059756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.059724    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 07:53:40.060378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.060362    4515 cpu_manager.go:414] "containerMap: removing container" podUID="59dc47c6-8416-4378-920b-af25c69f334f" containerName="maas-api"
Nov 02 07:53:40.060447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.060380    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dc47c6-8416-4378-920b-af25c69f334f" containerName="maas-api"
Nov 02 07:53:40.060492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.060483    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dc47c6-8416-4378-920b-af25c69f334f" containerName="maas-api"
Nov 02 07:53:40.064093 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.064076    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.088848 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.088819    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 07:53:40.095890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.095864    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7f19e8-openshift-default-dockercfg-5ss5l\""
Nov 02 07:53:40.163435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.163389    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:53:40.163678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.163653    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy" containerID="cri-o://2dfec31421f235b4ba1ed3adecae721ba9204e5983ed05a1cb2054630b67c45d" gracePeriod=30
Nov 02 07:53:40.170291 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.170256    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:40.183709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.183679    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:53:40.183952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.183928    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy" containerID="cri-o://64df4c7e23534a80cc21091e16cd87a777ea93947ac351bd441594033b26dd51" gracePeriod=30
Nov 02 07:53:40.191910 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.191882    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:40.192098 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.192076    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy" containerID="cri-o://ae585f84030ca06002152d1849f7a9356a65eb9c6c50da7772af1d024d6f0d23" gracePeriod=30
Nov 02 07:53:40.215078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215029    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215097    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215137    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215294    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215356    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215410    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215474    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddpl\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215575 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215557    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.215623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.215611    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.322620    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.322698    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.322754    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.322854    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddpl\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.322933    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.323001    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.323040    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.323083    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.323380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.323107    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.329112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.324427    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.329112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.328172    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.331094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.331068    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.331632 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.331349    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.331761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.331580    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.374458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.353054    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.374458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.365639    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.374458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.366879    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.374458 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.367633    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddpl\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\") pod \"e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") " pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.383712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.383418    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:40.546334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.539457    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy" containerID="cri-o://0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269" gracePeriod=30
Nov 02 07:53:40.913156 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:40.913119    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159a0cec_6643_43c9_9e64_33a38e868887.slice/crio-2ce524cede2d2d4023aac62a6a9baf960cb330ffd9c18767683d645ba89e17ec WatchSource:0}: Error finding container 2ce524cede2d2d4023aac62a6a9baf960cb330ffd9c18767683d645ba89e17ec: Status 404 returned error can't find the container with id 2ce524cede2d2d4023aac62a6a9baf960cb330ffd9c18767683d645ba89e17ec
Nov 02 07:53:40.915649 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.915626    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 07:53:40.916511 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.916476    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:40.916611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.916540    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:40.916611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:40.916569    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:41.546999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.546958    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerStarted","Data":"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"}
Nov 02 07:53:41.547409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.547028    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerStarted","Data":"2ce524cede2d2d4023aac62a6a9baf960cb330ffd9c18767683d645ba89e17ec"}
Nov 02 07:53:41.622721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.622667    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podStartSLOduration=1.6226527389999998 podStartE2EDuration="1.622652739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:41.606478522 +0000 UTC m=+1766.368915428" watchObservedRunningTime="2026-11-02 07:53:41.622652739 +0000 UTC m=+1766.385089644"
Nov 02 07:53:41.726165 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.726127    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 07:53:41.731186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.731156    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.749636 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.749610    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-56bbb6-openshift-default-dockercfg-sxfgp\""
Nov 02 07:53:41.759094 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.759049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 07:53:41.861982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.861901    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862012    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862133    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862171    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862195    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862274    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862297    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsqb\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862428 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862414    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.862474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.862459    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.967445    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.968843    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.968589    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973336    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973390    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973485    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973522    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsqb\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973555    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973650    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.976337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.973963    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.981331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.977605    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.981331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.977710    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:41.981331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:41.978412    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.012342 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.011804    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.016333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.013123    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.040390 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.032994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.053368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.046048    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.053368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.046801    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsqb\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\") pod \"e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") " pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.077333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.076601    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:42.384830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.384747    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:42.390590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.390564    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:42.453930 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:42.453898    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5237e1b_4892_48aa_8b09_d06e90fbd505.slice/crio-e7195b6d607f28493747d5d3dacfed4bc5e85fba9209214d6e41f9de93d8ee39 WatchSource:0}: Error finding container e7195b6d607f28493747d5d3dacfed4bc5e85fba9209214d6e41f9de93d8ee39: Status 404 returned error can't find the container with id e7195b6d607f28493747d5d3dacfed4bc5e85fba9209214d6e41f9de93d8ee39
Nov 02 07:53:42.454855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.454831    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 07:53:42.457060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.457026    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:42.457163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.457104    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:42.457163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.457147    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:42.556381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.556344    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerStarted","Data":"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"}
Nov 02 07:53:42.556793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.556405    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerStarted","Data":"e7195b6d607f28493747d5d3dacfed4bc5e85fba9209214d6e41f9de93d8ee39"}
Nov 02 07:53:42.556793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.556673    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:42.557714 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.557695    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:53:42.607885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:42.607816    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podStartSLOduration=1.6077946779999999 podStartE2EDuration="1.607794678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:42.602407497 +0000 UTC m=+1767.364844403" watchObservedRunningTime="2026-11-02 07:53:42.607794678 +0000 UTC m=+1767.370231585"
Nov 02 07:53:43.077422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:43.077392    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:43.082842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:43.082813    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:43.563551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:43.563522    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:43.564461 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:43.564444    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:53:45.632613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.632573    4515 generic.go:356] "Generic (PLEG): container finished" podID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerID="2dfec31421f235b4ba1ed3adecae721ba9204e5983ed05a1cb2054630b67c45d" exitCode=0
Nov 02 07:53:45.633078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.632731    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" event={"ID":"1fefbdc1-8411-4270-9814-120c80af3dd9","Type":"ContainerDied","Data":"2dfec31421f235b4ba1ed3adecae721ba9204e5983ed05a1cb2054630b67c45d"}
Nov 02 07:53:45.634973 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.634946    4515 generic.go:356] "Generic (PLEG): container finished" podID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerID="ae585f84030ca06002152d1849f7a9356a65eb9c6c50da7772af1d024d6f0d23" exitCode=0
Nov 02 07:53:45.635154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.635068    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" event={"ID":"8e1d9f7b-82af-4188-8148-723abaa4abf6","Type":"ContainerDied","Data":"ae585f84030ca06002152d1849f7a9356a65eb9c6c50da7772af1d024d6f0d23"}
Nov 02 07:53:45.638175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.638152    4515 generic.go:356] "Generic (PLEG): container finished" podID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerID="64df4c7e23534a80cc21091e16cd87a777ea93947ac351bd441594033b26dd51" exitCode=0
Nov 02 07:53:45.639009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.638345    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerDied","Data":"64df4c7e23534a80cc21091e16cd87a777ea93947ac351bd441594033b26dd51"}
Nov 02 07:53:45.639125 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.639059    4515 scope.go:122] "RemoveContainer" containerID="4614cfe98c4f9506db47ad84367cb91a0a61c76d814a7d5bbfc012c790a294d0"
Nov 02 07:53:45.728364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.720927    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:53:45.734752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.734663    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:53:45.743523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.743491    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.743968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.743904    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.744090 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.743981    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.744959 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.743571    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745113    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745189    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745245    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745295    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745738    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.746024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.745980    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.746679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.746506    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747134    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747186    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747257    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747288    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747905    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fefbdc1-8411-4270-9814-120c80af3dd9-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747926    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747939    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747952    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.748394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.747966    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.751389 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.751095    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.753403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.752569    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.753403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.752936    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "kube-api-access-9qx4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.753403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.752974    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "kube-api-access-kddtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.755097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.755063    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.782725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.782656    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:45.849330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.849285    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.849574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.849556    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.849689 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.849674    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.849897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.849882    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.850016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.850003    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\") pod \"1fefbdc1-8411-4270-9814-120c80af3dd9\" (UID: \"1fefbdc1-8411-4270-9814-120c80af3dd9\") "
Nov 02 07:53:45.850186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.850172    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.850465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.850447    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.850593 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.850578    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\") pod \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\" (UID: \"7fdd76f7-d16a-4d74-bdb6-673f5495ba54\") "
Nov 02 07:53:45.851370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851348    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qx4v\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-kube-api-access-9qx4v\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.851490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851477    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.851613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851591    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddtb\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-kube-api-access-kddtb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.851741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851617    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.851741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851636    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.851741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851501    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.851741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.851725    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.852183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.852160    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.852241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.852219    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.856549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.856478    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.857070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.857048    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.858071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.858050    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo" pod "7fdd76f7-d16a-4d74-bdb6-673f5495ba54" (UID: "7fdd76f7-d16a-4d74-bdb6-673f5495ba54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.858972 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.858949    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo" pod "1fefbdc1-8411-4270-9814-120c80af3dd9" (UID: "1fefbdc1-8411-4270-9814-120c80af3dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.922871 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.922841    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:45.952059 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952029    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952099    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952142    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952205    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952252 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952239    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952289    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952380    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952381    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952430    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952465    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952534    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\") pod \"8e1d9f7b-82af-4188-8148-723abaa4abf6\" (UID: \"8e1d9f7b-82af-4188-8148-723abaa4abf6\") "
Nov 02 07:53:45.952836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.952660    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.953214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953187    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.953214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953203    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953276    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d9f7b-82af-4188-8148-723abaa4abf6-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953293    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953303    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953331    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953340    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953348    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953356    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fefbdc1-8411-4270-9814-120c80af3dd9-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953365    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fefbdc1-8411-4270-9814-120c80af3dd9-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953374    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd76f7-d16a-4d74-bdb6-673f5495ba54-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953382    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.953406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.953389    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.954712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.954694    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.955046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.955016    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.955158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.955066    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "kube-api-access-xm67s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.955214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:45.955154    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy" pod "8e1d9f7b-82af-4188-8148-723abaa4abf6" (UID: "8e1d9f7b-82af-4188-8148-723abaa4abf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.054051 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054011    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054085    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054122    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054180    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054354    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054407    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.054560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054425    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054602    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054707    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054737    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\") pod \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\" (UID: \"53b86f17-ddc4-4e6b-9ed1-dd232c724260\") "
Nov 02 07:53:46.054913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.054767    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.055104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055079    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055335    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055556    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055570    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055580    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055598    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.055616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055614    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055628    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055670    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055686    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm67s\" (UniqueName: \"kubernetes.io/projected/8e1d9f7b-82af-4188-8148-723abaa4abf6-kube-api-access-xm67s\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055702    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d9f7b-82af-4188-8148-723abaa4abf6-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055717    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056006 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.055736    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d9f7b-82af-4188-8148-723abaa4abf6-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.056865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.056841    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:46.056943 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.056871    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:46.056992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.056949    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:46.057036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.057018    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh" pod "53b86f17-ddc4-4e6b-9ed1-dd232c724260" (UID: "53b86f17-ddc4-4e6b-9ed1-dd232c724260"). InnerVolumeSpecName "kube-api-access-kcvwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:46.157817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.157720    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.157817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.157760    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.157817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.157776    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.157817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.157789    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvwh\" (UniqueName: \"kubernetes.io/projected/53b86f17-ddc4-4e6b-9ed1-dd232c724260-kube-api-access-kcvwh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.157817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.157805    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b86f17-ddc4-4e6b-9ed1-dd232c724260-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.316863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.316830    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 07:53:46.317497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317481    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317499    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317512    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317517    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317534    4515 cpu_manager.go:414] "containerMap: removing container" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy"
Nov 02 07:53:46.317541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317539    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317556    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317560    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317568    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317573    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317660    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317669    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317677    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317686    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" containerName="istio-proxy"
Nov 02 07:53:46.317698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.317691    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" containerName="istio-proxy"
Nov 02 07:53:46.322677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.322651    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.338531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.338504    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-7f19e8\""
Nov 02 07:53:46.339901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.339869    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 07:53:46.361181 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.361146    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwt4\" (UniqueName: \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.361492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.361469    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.409609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.392845    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-7f19e8-dockercfg-gxvrb\""
Nov 02 07:53:46.467754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.454946    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 07:53:46.468846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.468806    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwt4\" (UniqueName: \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.470840 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.470811    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.472263 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.472237    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.473108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.470212    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.477975 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.477192    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 07:53:46.547955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.541986    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwt4\" (UniqueName: \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\") pod \"payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.639155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.639113    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:46.647102 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.647055    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"
Nov 02 07:53:46.647258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.647076    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf" event={"ID":"8e1d9f7b-82af-4188-8148-723abaa4abf6","Type":"ContainerDied","Data":"c94c9017bfd54d4b17d539528fc791937a1ff810af16b2bec5b0650411eb0858"}
Nov 02 07:53:46.647258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.647249    4515 scope.go:122] "RemoveContainer" containerID="ae585f84030ca06002152d1849f7a9356a65eb9c6c50da7772af1d024d6f0d23"
Nov 02 07:53:46.648993 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.648966    4515 generic.go:356] "Generic (PLEG): container finished" podID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" containerID="0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269" exitCode=0
Nov 02 07:53:46.649104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.649025    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"
Nov 02 07:53:46.649104 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.649047    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" event={"ID":"53b86f17-ddc4-4e6b-9ed1-dd232c724260","Type":"ContainerDied","Data":"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"}
Nov 02 07:53:46.649222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.649102    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r" event={"ID":"53b86f17-ddc4-4e6b-9ed1-dd232c724260","Type":"ContainerDied","Data":"5593d79573e504e6f10a1d1dc8b0d8a48eb109e6d695db66db3fbad9ed33a031"}
Nov 02 07:53:46.653025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.652996    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"
Nov 02 07:53:46.653025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.653013    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx" event={"ID":"7fdd76f7-d16a-4d74-bdb6-673f5495ba54","Type":"ContainerDied","Data":"31a6887765a0150e05f48bd17ca674f07cf5ea235d51eaa50c430dbdddabe29e"}
Nov 02 07:53:46.654719 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.654697    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q" event={"ID":"1fefbdc1-8411-4270-9814-120c80af3dd9","Type":"ContainerDied","Data":"9fc386a6c0c22e7200d816abea11a4b355b6f23826c5c8210a29f6f9970c4733"}
Nov 02 07:53:46.654810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.654789    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"
Nov 02 07:53:46.685730 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.685679    4515 scope.go:122] "RemoveContainer" containerID="0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"
Nov 02 07:53:46.699403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.699374    4515 scope.go:122] "RemoveContainer" containerID="0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"
Nov 02 07:53:46.699752 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:46.699718    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269\": container with ID starting with 0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269 not found: ID does not exist" containerID="0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"
Nov 02 07:53:46.699883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.699764    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269"} err="failed to get container status \"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269\": rpc error: code = NotFound desc = could not find container \"0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269\": container with ID starting with 0c52cf78f3cc0779919ef76eb9d983504ad8da25f243e0b7ed914c04f1817269 not found: ID does not exist"
Nov 02 07:53:46.699883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.699790    4515 scope.go:122] "RemoveContainer" containerID="64df4c7e23534a80cc21091e16cd87a777ea93947ac351bd441594033b26dd51"
Nov 02 07:53:46.700044 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.700020    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.700338 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.700300    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx5p\" (UniqueName: \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.712865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.712831    4515 scope.go:122] "RemoveContainer" containerID="2dfec31421f235b4ba1ed3adecae721ba9204e5983ed05a1cb2054630b67c45d"
Nov 02 07:53:46.759561 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.759518    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:53:46.764795 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.764761    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-6rhpx"]
Nov 02 07:53:46.800129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.800094    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:46.801949 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.801925    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx5p\" (UniqueName: \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.802042 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.802021    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.802718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.802700    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.803053 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.803034    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-gjx6r"]
Nov 02 07:53:46.826259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.826231    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx5p\" (UniqueName: \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\") pod \"payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.838449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.838426    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:46.841226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.841206    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-qbdvf"]
Nov 02 07:53:46.861413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.861386    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:46.877835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.877807    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:53:46.884739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.884714    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lrr2q"]
Nov 02 07:53:46.959295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:46.959264    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 07:53:47.168008 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:47.167980    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188801de_30fb_4c45_8c84_adb7d9979a00.slice/crio-efd94b40bb4236c5e5e05119675cc932f71c3b30fc4f3cc4028b32b010b8916b WatchSource:0}: Error finding container efd94b40bb4236c5e5e05119675cc932f71c3b30fc4f3cc4028b32b010b8916b: Status 404 returned error can't find the container with id efd94b40bb4236c5e5e05119675cc932f71c3b30fc4f3cc4028b32b010b8916b
Nov 02 07:53:47.168902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.168879    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 07:53:47.374618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.374583    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fefbdc1-8411-4270-9814-120c80af3dd9" path="/var/lib/kubelet/pods/1fefbdc1-8411-4270-9814-120c80af3dd9/volumes"
Nov 02 07:53:47.375192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.375167    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b86f17-ddc4-4e6b-9ed1-dd232c724260" path="/var/lib/kubelet/pods/53b86f17-ddc4-4e6b-9ed1-dd232c724260/volumes"
Nov 02 07:53:47.375710 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.375692    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdd76f7-d16a-4d74-bdb6-673f5495ba54" path="/var/lib/kubelet/pods/7fdd76f7-d16a-4d74-bdb6-673f5495ba54/volumes"
Nov 02 07:53:47.376605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.376587    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1d9f7b-82af-4188-8148-723abaa4abf6" path="/var/lib/kubelet/pods/8e1d9f7b-82af-4188-8148-723abaa4abf6/volumes"
Nov 02 07:53:47.667026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.666982    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" event={"ID":"188801de-30fb-4c45-8c84-adb7d9979a00","Type":"ContainerStarted","Data":"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"}
Nov 02 07:53:47.667616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.667058    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" event={"ID":"188801de-30fb-4c45-8c84-adb7d9979a00","Type":"ContainerStarted","Data":"efd94b40bb4236c5e5e05119675cc932f71c3b30fc4f3cc4028b32b010b8916b"}
Nov 02 07:53:47.667616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.667267    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:47.668880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.668852    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" event={"ID":"939815fc-f120-4acb-a238-3d2740dff156","Type":"ContainerStarted","Data":"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"}
Nov 02 07:53:47.669013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.668898    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" event={"ID":"939815fc-f120-4acb-a238-3d2740dff156","Type":"ContainerStarted","Data":"e46f94475d4569ac507c2b1422200233f6ea7a6309c9551915ddbc2b27e188b8"}
Nov 02 07:53:47.669013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.668922    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:47.709704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.709660    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" podStartSLOduration=1.7096481300000002 podStartE2EDuration="1.70964813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:47.709401347 +0000 UTC m=+1772.471838252" watchObservedRunningTime="2026-11-02 07:53:47.70964813 +0000 UTC m=+1772.472085035"
Nov 02 07:53:47.723697 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:47.723668    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:53:47.752656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:47.752609    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" podStartSLOduration=1.752585703 podStartE2EDuration="1.752585703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:47.750796237 +0000 UTC m=+1772.513233146" watchObservedRunningTime="2026-11-02 07:53:47.752585703 +0000 UTC m=+1772.515022608"
Nov 02 07:53:48.294741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.294707    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 07:53:48.300455 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.300431    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.316078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.316048    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-32ea65-dockercfg-htcsh\""
Nov 02 07:53:48.319732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.319696    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 07:53:48.337345 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.337303    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-32ea65\""
Nov 02 07:53:48.431395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.431354    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.431726 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.431627    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgnv\" (UniqueName: \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.534332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.533601    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.534332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.533760    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgnv\" (UniqueName: \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.538330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.535057    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.588652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.586125    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgnv\" (UniqueName: \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\") pod \"payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.623936 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.622883    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:48.708394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:48.707839    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:53:49.093902 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:53:49.093864    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11c1b49_75dd_4ee4_ba46_af5a755cb572.slice/crio-fa9aecbceb138e4a3626f7e6e47f10119b52900e0c2fb1fcc205fe8a83edb41d WatchSource:0}: Error finding container fa9aecbceb138e4a3626f7e6e47f10119b52900e0c2fb1fcc205fe8a83edb41d: Status 404 returned error can't find the container with id fa9aecbceb138e4a3626f7e6e47f10119b52900e0c2fb1fcc205fe8a83edb41d
Nov 02 07:53:49.095405 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:49.095380    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 07:53:49.715989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:49.715950    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" event={"ID":"b11c1b49-75dd-4ee4-ba46-af5a755cb572","Type":"ContainerStarted","Data":"f91694e4a23b7bca33f5abf7e9e17ea5949bcb7869ffd4d4d0c532e4286863ab"}
Nov 02 07:53:49.716436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:49.716008    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" event={"ID":"b11c1b49-75dd-4ee4-ba46-af5a755cb572","Type":"ContainerStarted","Data":"fa9aecbceb138e4a3626f7e6e47f10119b52900e0c2fb1fcc205fe8a83edb41d"}
Nov 02 07:53:49.716436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:49.716424    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:53:49.759761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:49.759703    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" podStartSLOduration=1.759690617 podStartE2EDuration="1.759690617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:49.758412189 +0000 UTC m=+1774.520849096" watchObservedRunningTime="2026-11-02 07:53:49.759690617 +0000 UTC m=+1774.522127523"
Nov 02 07:53:52.908109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:52.908074    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 07:53:52.912826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:52.912803    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:52.942335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:52.933386    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e4c131-openshift-default-dockercfg-m59kq\""
Nov 02 07:53:52.974129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:52.974100    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 07:53:53.011474 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011429    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqptz\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011504    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011610    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011646    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011834 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011777    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.011952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011893    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.012005 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.011966    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.012062 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.012017    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.112855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.112817    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.112855 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.112887    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.112991    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqptz\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113079    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113108    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113133    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113206    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113250    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113302    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.113712 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.113488    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.114078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.114050    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.114194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.114125    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.114194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.114141    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.149333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.148184    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.149333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.148766    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.155933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.155867    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.155933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.155870    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqptz\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\") pod \"e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") " pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.279968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.279925    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:53.844236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.844201    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 07:53:53.847981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.847930    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:53.848077 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.848028    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:53.848117 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:53.848096    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:54.651281 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.651230    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:53:54.652092 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.652047    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" podUID="6378e1c5-9db9-44ba-bb05-304c910a1b61" containerName="maas-api" containerID="cri-o://195edf8279ad356174c2df6b4dede79661a7a65a2935123fae1a8dd29ce97428" gracePeriod=30
Nov 02 07:53:54.777804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.777774    4515 generic.go:356] "Generic (PLEG): container finished" podID="6378e1c5-9db9-44ba-bb05-304c910a1b61" containerID="195edf8279ad356174c2df6b4dede79661a7a65a2935123fae1a8dd29ce97428" exitCode=0
Nov 02 07:53:54.777952 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.777847    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" event={"ID":"6378e1c5-9db9-44ba-bb05-304c910a1b61","Type":"ContainerDied","Data":"195edf8279ad356174c2df6b4dede79661a7a65a2935123fae1a8dd29ce97428"}
Nov 02 07:53:54.779863 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.779830    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerStarted","Data":"1efd6b2fdf359b1dc9f1372b7f1d87a67d3c80e65bb2a9a7610bd1b4c4802d7c"}
Nov 02 07:53:54.780011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.779892    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerStarted","Data":"38b96ec18173186a92c6668667d4936b894b8d65db4e0b0a30e72ca2c342e242"}
Nov 02 07:53:54.852543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.849677    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:53:54.852543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.849911    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" podUID="903e89cc-a05d-41cf-941e-539b23a06e4a" containerName="payload-processing" containerID="cri-o://91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076" gracePeriod=30
Nov 02 07:53:54.854020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.853967    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podStartSLOduration=2.8539494 podStartE2EDuration="2.8539494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:54.843803859 +0000 UTC m=+1779.606240767" watchObservedRunningTime="2026-11-02 07:53:54.8539494 +0000 UTC m=+1779.616386307"
Nov 02 07:53:54.869974 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.869942    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:53:54.870302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:54.870239    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" podUID="ddd46948-365f-4741-b531-5d9b598daeff" containerName="payload-pre-processing" containerID="cri-o://4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3" gracePeriod=30
Nov 02 07:53:55.052825 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.052758    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:53:55.190326 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.190271    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\") pod \"6378e1c5-9db9-44ba-bb05-304c910a1b61\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") "
Nov 02 07:53:55.190500 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.190479    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\") pod \"6378e1c5-9db9-44ba-bb05-304c910a1b61\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") "
Nov 02 07:53:55.191048 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.191020    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\" (UniqueName: \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\") pod \"6378e1c5-9db9-44ba-bb05-304c910a1b61\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") "
Nov 02 07:53:55.191226 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.191207    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\") pod \"6378e1c5-9db9-44ba-bb05-304c910a1b61\" (UID: \"6378e1c5-9db9-44ba-bb05-304c910a1b61\") "
Nov 02 07:53:55.192553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.192128    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca" pod "6378e1c5-9db9-44ba-bb05-304c910a1b61" (UID: "6378e1c5-9db9-44ba-bb05-304c910a1b61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:55.193847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.193820    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6378e1c5-9db9-44ba-bb05-304c910a1b61-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.204531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.204453    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls" pod "6378e1c5-9db9-44ba-bb05-304c910a1b61" (UID: "6378e1c5-9db9-44ba-bb05-304c910a1b61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:55.206761 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.206431    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs" pod "6378e1c5-9db9-44ba-bb05-304c910a1b61" (UID: "6378e1c5-9db9-44ba-bb05-304c910a1b61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:55.208931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.208902    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6" pod "6378e1c5-9db9-44ba-bb05-304c910a1b61" (UID: "6378e1c5-9db9-44ba-bb05-304c910a1b61"). InnerVolumeSpecName "kube-api-access-wdmf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:55.287352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.286755    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:55.458400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.449003    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmf6\" (UniqueName: \"kubernetes.io/projected/6378e1c5-9db9-44ba-bb05-304c910a1b61-kube-api-access-wdmf6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.458400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.449044    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.458400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.449059    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6378e1c5-9db9-44ba-bb05-304c910a1b61-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.458400 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.457248    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:55.481725 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.481277    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:53:55.484361 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.484287    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:53:55.549775 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.549741    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\") pod \"903e89cc-a05d-41cf-941e-539b23a06e4a\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") "
Nov 02 07:53:55.549964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.549865    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\") pod \"ddd46948-365f-4741-b531-5d9b598daeff\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") "
Nov 02 07:53:55.549964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.549916    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\" (UniqueName: \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\") pod \"903e89cc-a05d-41cf-941e-539b23a06e4a\" (UID: \"903e89cc-a05d-41cf-941e-539b23a06e4a\") "
Nov 02 07:53:55.550075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.549999    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\" (UniqueName: \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\") pod \"ddd46948-365f-4741-b531-5d9b598daeff\" (UID: \"ddd46948-365f-4741-b531-5d9b598daeff\") "
Nov 02 07:53:55.550254 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.550226    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume" pod "903e89cc-a05d-41cf-941e-539b23a06e4a" (UID: "903e89cc-a05d-41cf-941e-539b23a06e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:55.550364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.550238    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume" pod "ddd46948-365f-4741-b531-5d9b598daeff" (UID: "ddd46948-365f-4741-b531-5d9b598daeff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:55.550570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.550553    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd46948-365f-4741-b531-5d9b598daeff-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.550644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.550573    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903e89cc-a05d-41cf-941e-539b23a06e4a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.552786 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.552765    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74" pod "903e89cc-a05d-41cf-941e-539b23a06e4a" (UID: "903e89cc-a05d-41cf-941e-539b23a06e4a"). InnerVolumeSpecName "kube-api-access-qhk74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:55.552877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.552793    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7" pod "ddd46948-365f-4741-b531-5d9b598daeff" (UID: "ddd46948-365f-4741-b531-5d9b598daeff"). InnerVolumeSpecName "kube-api-access-4bfb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:55.652159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.652120    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhk74\" (UniqueName: \"kubernetes.io/projected/903e89cc-a05d-41cf-941e-539b23a06e4a-kube-api-access-qhk74\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.652159 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.652151    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfb7\" (UniqueName: \"kubernetes.io/projected/ddd46948-365f-4741-b531-5d9b598daeff-kube-api-access-4bfb7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:53:55.800331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.799887    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55" event={"ID":"6378e1c5-9db9-44ba-bb05-304c910a1b61","Type":"ContainerDied","Data":"5cce0eb739dbb8e42f62fef3b786d590a92b6c497abc11e1eea72f1d85534b36"}
Nov 02 07:53:55.800331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.799964    4515 scope.go:122] "RemoveContainer" containerID="195edf8279ad356174c2df6b4dede79661a7a65a2935123fae1a8dd29ce97428"
Nov 02 07:53:55.800331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.800134    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"
Nov 02 07:53:55.804550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.804500    4515 generic.go:356] "Generic (PLEG): container finished" podID="903e89cc-a05d-41cf-941e-539b23a06e4a" containerID="91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076" exitCode=0
Nov 02 07:53:55.804677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.804600    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" event={"ID":"903e89cc-a05d-41cf-941e-539b23a06e4a","Type":"ContainerDied","Data":"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"}
Nov 02 07:53:55.804677 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.804651    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt" event={"ID":"903e89cc-a05d-41cf-941e-539b23a06e4a","Type":"ContainerDied","Data":"e02a534601298721c0d0d6586b05a72df4599f0e5c10cdcdcb69d7ca62785908"}
Nov 02 07:53:55.804763 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.804738    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"
Nov 02 07:53:55.808670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.808586    4515 generic.go:356] "Generic (PLEG): container finished" podID="ddd46948-365f-4741-b531-5d9b598daeff" containerID="4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3" exitCode=0
Nov 02 07:53:55.808670 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.808638    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"
Nov 02 07:53:55.808858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.808680    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" event={"ID":"ddd46948-365f-4741-b531-5d9b598daeff","Type":"ContainerDied","Data":"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"}
Nov 02 07:53:55.808858 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.808746    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n" event={"ID":"ddd46948-365f-4741-b531-5d9b598daeff","Type":"ContainerDied","Data":"c156eb5bc6d399e76b8dddb8111003a1d80d924c605966c87ea2ace5f332cab0"}
Nov 02 07:53:55.810019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.809982    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:55.831379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.812343    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:53:55.831379 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.827796    4515 scope.go:122] "RemoveContainer" containerID="91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"
Nov 02 07:53:55.845784 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.845763    4515 scope.go:122] "RemoveContainer" containerID="91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"
Nov 02 07:53:55.846135 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:55.846113    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076\": container with ID starting with 91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076 not found: ID does not exist" containerID="91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"
Nov 02 07:53:55.846209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.846146    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076"} err="failed to get container status \"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076\": rpc error: code = NotFound desc = could not find container \"91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076\": container with ID starting with 91ae74b160bb7a7afa871a48d54986e258d1c1195c6c98ab4938dafebb6a0076 not found: ID does not exist"
Nov 02 07:53:55.846209 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.846165    4515 scope.go:122] "RemoveContainer" containerID="4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"
Nov 02 07:53:55.856004 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.855978    4515 scope.go:122] "RemoveContainer" containerID="4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"
Nov 02 07:53:55.856290 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:53:55.856273    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3\": container with ID starting with 4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3 not found: ID does not exist" containerID="4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"
Nov 02 07:53:55.856381 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.856298    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3"} err="failed to get container status \"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3\": rpc error: code = NotFound desc = could not find container \"4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3\": container with ID starting with 4608a52427513b77a56ce0295f4223bc666e38600533d08ae1f78ec6ed301ac3 not found: ID does not exist"
Nov 02 07:53:55.873277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.873251    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:53:55.880091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.880069    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-16e21e-9bb6f6d85-stn55"]
Nov 02 07:53:55.978894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.978809    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:53:55.983981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:55.983939    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-16e21e-6f656f9df4-chqqt"]
Nov 02 07:53:56.024201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:56.024172    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:53:56.031590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:56.031566    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-16e21e-5b99c4d595-ntr8n"]
Nov 02 07:53:57.376894 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:57.376853    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6378e1c5-9db9-44ba-bb05-304c910a1b61" path="/var/lib/kubelet/pods/6378e1c5-9db9-44ba-bb05-304c910a1b61/volumes"
Nov 02 07:53:57.377489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:57.377471    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903e89cc-a05d-41cf-941e-539b23a06e4a" path="/var/lib/kubelet/pods/903e89cc-a05d-41cf-941e-539b23a06e4a/volumes"
Nov 02 07:53:57.377960 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:57.377946    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd46948-365f-4741-b531-5d9b598daeff" path="/var/lib/kubelet/pods/ddd46948-365f-4741-b531-5d9b598daeff/volumes"
Nov 02 07:53:58.712112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:58.712079    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 07:53:58.712620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:58.712154    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 07:53:59.474812 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.474774    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 07:53:59.475557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475536    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6378e1c5-9db9-44ba-bb05-304c910a1b61" containerName="maas-api"
Nov 02 07:53:59.475686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475559    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6378e1c5-9db9-44ba-bb05-304c910a1b61" containerName="maas-api"
Nov 02 07:53:59.475686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475597    4515 cpu_manager.go:414] "containerMap: removing container" podUID="903e89cc-a05d-41cf-941e-539b23a06e4a" containerName="payload-processing"
Nov 02 07:53:59.475686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475605    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903e89cc-a05d-41cf-941e-539b23a06e4a" containerName="payload-processing"
Nov 02 07:53:59.475686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475636    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ddd46948-365f-4741-b531-5d9b598daeff" containerName="payload-pre-processing"
Nov 02 07:53:59.475686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475642    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd46948-365f-4741-b531-5d9b598daeff" containerName="payload-pre-processing"
Nov 02 07:53:59.476019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475755    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6378e1c5-9db9-44ba-bb05-304c910a1b61" containerName="maas-api"
Nov 02 07:53:59.476019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475765    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="903e89cc-a05d-41cf-941e-539b23a06e4a" containerName="payload-processing"
Nov 02 07:53:59.476019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.475773    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd46948-365f-4741-b531-5d9b598daeff" containerName="payload-pre-processing"
Nov 02 07:53:59.478738 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.478717    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.494835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.494809    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e4c131-dockercfg-l5s2h\""
Nov 02 07:53:59.498003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.497981    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 07:53:59.506541 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.506520    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e4c131\""
Nov 02 07:53:59.542144 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.542074    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 07:53:59.546768 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.546749    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.562941 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.562921    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 07:53:59.655154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.630721    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2nh\" (UniqueName: \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.655154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.630792    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.655154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.630825    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.655154 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.630902    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw72c\" (UniqueName: \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.735418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.734269    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw72c\" (UniqueName: \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.735418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.734589    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2nh\" (UniqueName: \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.735418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.734641    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.735418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.734670    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.736112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.735605    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.740337 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.736752    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.781700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.777202    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2nh\" (UniqueName: \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\") pod \"payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.781700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.777643    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw72c\" (UniqueName: \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\") pod \"payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:53:59.802336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.801653    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:53:59.853919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.853884    4515 generic.go:356] "Generic (PLEG): container finished" podID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerID="7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c" exitCode=0
Nov 02 07:53:59.854095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.853952    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerDied","Data":"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"}
Nov 02 07:53:59.854436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.854417    4515 scope.go:122] "RemoveContainer" containerID="7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"
Nov 02 07:53:59.856161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.856130    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:59.856257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.856215    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:59.856257 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.856250    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:59.862787 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:53:59.861105    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:54:00.145799 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.145769    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 07:54:00.145935 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:00.145904    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd1a400_35fd_439a_9f00_105658388f10.slice/crio-4f243b001ae8827c7acc8219eb87c8261f2251e6e653e2cc8c2587b43607b6ca WatchSource:0}: Error finding container 4f243b001ae8827c7acc8219eb87c8261f2251e6e653e2cc8c2587b43607b6ca: Status 404 returned error can't find the container with id 4f243b001ae8827c7acc8219eb87c8261f2251e6e653e2cc8c2587b43607b6ca
Nov 02 07:54:00.196396 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:00.196366    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d2b70c_a401_4493_bac9_fd92f603ca7b.slice/crio-e172585907ca640f2e26d520347267d1ca6b15851be15ce9cd96ee1406afa6da WatchSource:0}: Error finding container e172585907ca640f2e26d520347267d1ca6b15851be15ce9cd96ee1406afa6da: Status 404 returned error can't find the container with id e172585907ca640f2e26d520347267d1ca6b15851be15ce9cd96ee1406afa6da
Nov 02 07:54:00.197286 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.197257    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 07:54:00.748480 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.748450    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 07:54:00.864295 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.864259    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" event={"ID":"63d2b70c-a401-4493-bac9-fd92f603ca7b","Type":"ContainerStarted","Data":"35193ee2ecc522070c9d54d1e51722c91d4545c7b1360e57f565d0c002d6b170"}
Nov 02 07:54:00.864495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.864347    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" event={"ID":"63d2b70c-a401-4493-bac9-fd92f603ca7b","Type":"ContainerStarted","Data":"e172585907ca640f2e26d520347267d1ca6b15851be15ce9cd96ee1406afa6da"}
Nov 02 07:54:00.865176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.865157    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:54:00.866923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.866897    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" event={"ID":"0cd1a400-35fd-439a-9f00-105658388f10","Type":"ContainerStarted","Data":"55ff935c12c2dd648b4ccb03ac13db1e82bbd76a7e7f87b6559e37f5be02a8f5"}
Nov 02 07:54:00.867049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.866938    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" event={"ID":"0cd1a400-35fd-439a-9f00-105658388f10","Type":"ContainerStarted","Data":"4f243b001ae8827c7acc8219eb87c8261f2251e6e653e2cc8c2587b43607b6ca"}
Nov 02 07:54:00.867509 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.867493    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:54:00.869264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.869243    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerStarted","Data":"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"}
Nov 02 07:54:00.913107 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:00.913053    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" podStartSLOduration=1.9130373600000001 podStartE2EDuration="1.91303736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:00.912115669 +0000 UTC m=+1785.674552577" watchObservedRunningTime="2026-11-02 07:54:00.91303736 +0000 UTC m=+1785.675474266"
Nov 02 07:54:01.043902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:01.040220    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:54:01.043902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:01.040366    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:01.043902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:01.040425    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:01.070334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:01.067026    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" podStartSLOduration=2.067004798 podStartE2EDuration="2.067004798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:01.014282085 +0000 UTC m=+1785.776719011" watchObservedRunningTime="2026-11-02 07:54:01.067004798 +0000 UTC m=+1785.829441706"
Nov 02 07:54:02.025332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:02.023708    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:02.025332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:02.023786    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:03.041380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.032732    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:03.041380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.032812    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:03.059248 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.059211    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 07:54:03.063711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.063689    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.079169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.079146    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0fbc17-openshift-default-dockercfg-p76f6\""
Nov 02 07:54:03.098354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.095495    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 07:54:03.216146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216102    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216177    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216220    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216263    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216347    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216410    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216473    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216527    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.216613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.216596    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.333045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.322156    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.358586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.333345    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359009    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359198    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.325239    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359352    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359618 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359603    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359726    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.359864 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.359848    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.360025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.360005    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.360116 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.360019    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.360280 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.360264    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.360726 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.360698    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.360899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.360882    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.363612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.363590    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.364917 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.364898    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.372142 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.372120    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.384491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.384469    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:03.689404 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:03.686579    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:04.024842 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.024802    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:04.025130 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.025093    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:04.150544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.150508    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 07:54:04.156180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.156151    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.161659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.161631    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 07:54:04.173780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.173733    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-7f19e8-serving-cert\""
Nov 02 07:54:04.215715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.215682    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.215880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.215734    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.215968 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.215935    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.216009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.215984    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr58\" (UniqueName: \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.340392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.317093    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.340392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.317166    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.340392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.317417    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.340392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.317489    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr58\" (UniqueName: \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.340392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.318700    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.343748 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.343706    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 07:54:04.345861 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.345841    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.346185 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.346159    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr58\" (UniqueName: \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.346239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.346219    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.349404 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:04.349366    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef95c73_f6ae_4db8_8987_00d772d9cd4c.slice/crio-b27cef6b2e1393015f2cd394f14ef8e7173e845a1977697d256bd42d6a283a5d WatchSource:0}: Error finding container b27cef6b2e1393015f2cd394f14ef8e7173e845a1977697d256bd42d6a283a5d: Status 404 returned error can't find the container with id b27cef6b2e1393015f2cd394f14ef8e7173e845a1977697d256bd42d6a283a5d
Nov 02 07:54:04.361810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.361415    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:04.363202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.362117    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:04.363202 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.362200    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:04.475957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.475600    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:04.812790 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:04.812760    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638f4e75_80bb_4d6c_b813_b5cbeaa1e155.slice/crio-1999f818bb584cac63bf70365c15c315dc94251b6b4847b1b425e74054b8f9f1 WatchSource:0}: Error finding container 1999f818bb584cac63bf70365c15c315dc94251b6b4847b1b425e74054b8f9f1: Status 404 returned error can't find the container with id 1999f818bb584cac63bf70365c15c315dc94251b6b4847b1b425e74054b8f9f1
Nov 02 07:54:04.814131 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:04.814103    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 07:54:05.027347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.025701    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:05.027347 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.025765    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:05.074335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.073904    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" event={"ID":"aef95c73-f6ae-4db8-8987-00d772d9cd4c","Type":"ContainerStarted","Data":"28135d849d64281d9eed4abe0fbb282b84068b22f2326e375679d0b16c3e126e"}
Nov 02 07:54:05.074335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.073997    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" event={"ID":"aef95c73-f6ae-4db8-8987-00d772d9cd4c","Type":"ContainerStarted","Data":"b27cef6b2e1393015f2cd394f14ef8e7173e845a1977697d256bd42d6a283a5d"}
Nov 02 07:54:05.087791 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.085579    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" event={"ID":"638f4e75-80bb-4d6c-b813-b5cbeaa1e155","Type":"ContainerStarted","Data":"1999f818bb584cac63bf70365c15c315dc94251b6b4847b1b425e74054b8f9f1"}
Nov 02 07:54:05.155612 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.145616    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podStartSLOduration=2.145594394 podStartE2EDuration="2.145594394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:05.120790628 +0000 UTC m=+1789.883227535" watchObservedRunningTime="2026-11-02 07:54:05.145594394 +0000 UTC m=+1789.908031305"
Nov 02 07:54:05.694685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.694591    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:05.696283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.696246    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:05.696418 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:05.696305    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:06.023760 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.023723    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:54:06.024057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.024028    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 07:54:06.095239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.095193    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" event={"ID":"638f4e75-80bb-4d6c-b813-b5cbeaa1e155","Type":"ContainerStarted","Data":"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"}
Nov 02 07:54:06.095421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.095294    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:06.160331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.159731    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" podStartSLOduration=1.703876493 podStartE2EDuration="2.159706201s" totalImagesPullingTime="455.829708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:06.139720834 +0000 UTC m=+1790.902157744" watchObservedRunningTime="2026-11-02 07:54:06.159706201 +0000 UTC m=+1790.922143108"
Nov 02 07:54:06.333503 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.331409    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 07:54:06.358343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.352690    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.383336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.372362    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 07:54:06.391881 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.391836    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-32ea65-serving-cert\""
Nov 02 07:54:06.458330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.457357    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.458330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.457627    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.458330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.457695    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlkr\" (UniqueName: \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.458330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.457833    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.560188 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.560146    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.560440 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.560421    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.561015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.560986    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.561122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.561015    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.561122 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.561041    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlkr\" (UniqueName: \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.563721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.563611    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.563866 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.563745    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.585648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.585560    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlkr\" (UniqueName: \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\") pod \"maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:06.698363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.694519    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:06.698363 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.694583    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:06.717339 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:06.717289    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:07.046687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.046654    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:54:07.047457 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.047411    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:54:07.048146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.048129    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 07:54:07.079693 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:07.079663    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3a587a_54db_4b3a_b2cb_62dbdcf91179.slice/crio-5f068459367b99fcf37119499832644eb9cecbd0255673b9021ad53589a27fd8 WatchSource:0}: Error finding container 5f068459367b99fcf37119499832644eb9cecbd0255673b9021ad53589a27fd8: Status 404 returned error can't find the container with id 5f068459367b99fcf37119499832644eb9cecbd0255673b9021ad53589a27fd8
Nov 02 07:54:07.082802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.082775    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 07:54:07.106328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.106229    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" event={"ID":"5c3a587a-54db-4b3a-b2cb-62dbdcf91179","Type":"ContainerStarted","Data":"5f068459367b99fcf37119499832644eb9cecbd0255673b9021ad53589a27fd8"}
Nov 02 07:54:07.476924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.476817    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:54:07.477459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.477115    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy" containerID="cri-o://0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09" gracePeriod=30
Nov 02 07:54:07.695926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.695892    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:07.696171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:07.696128    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:08.114180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.114142    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" event={"ID":"5c3a587a-54db-4b3a-b2cb-62dbdcf91179","Type":"ContainerStarted","Data":"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"}
Nov 02 07:54:08.166039 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.165976    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" podStartSLOduration=1.756955961 podStartE2EDuration="2.165960585s" totalImagesPullingTime="409.004624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:08.16125439 +0000 UTC m=+1792.923691295" watchObservedRunningTime="2026-11-02 07:54:08.165960585 +0000 UTC m=+1792.928397491"
Nov 02 07:54:08.694570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.694529    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:08.694995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.694593    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:08.746049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.746008    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 07:54:08.750344 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.750317    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:08.765901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.765872    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-56bbb6-serving-cert\""
Nov 02 07:54:08.767633 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.767608    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 07:54:08.939332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.939149    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvhd\" (UniqueName: \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:08.939544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.939366    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:08.939544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.939409    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:08.939544 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:08.939478    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.052330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.048951    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvhd\" (UniqueName: \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.052330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.049160    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.052330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.049279    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.059331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.053691    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.066716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.066683    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.083504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.081153    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.094368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.084628    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvhd\" (UniqueName: \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.117348 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.116189    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.149336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.148932    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:09.368724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.368641    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:09.694408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.694320    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:09.694408 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.694382    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:09.703377 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:09.703346    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54c8c31_e110_431c_8233_7e7222e45ddf.slice/crio-2cefc161c593b358d9363b219beb5ae8bf80cf2dc42af062008cc4bf4d9de9bc WatchSource:0}: Error finding container 2cefc161c593b358d9363b219beb5ae8bf80cf2dc42af062008cc4bf4d9de9bc: Status 404 returned error can't find the container with id 2cefc161c593b358d9363b219beb5ae8bf80cf2dc42af062008cc4bf4d9de9bc
Nov 02 07:54:09.704479 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:09.704453    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 07:54:10.245464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:10.245431    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" event={"ID":"e54c8c31-e110-431c-8233-7e7222e45ddf","Type":"ContainerStarted","Data":"2cefc161c593b358d9363b219beb5ae8bf80cf2dc42af062008cc4bf4d9de9bc"}
Nov 02 07:54:10.694919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:10.694832    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 07:54:10.694919 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:10.694887    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 07:54:11.265940 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:11.265902    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" event={"ID":"e54c8c31-e110-431c-8233-7e7222e45ddf","Type":"ContainerStarted","Data":"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"}
Nov 02 07:54:11.266530 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:11.266086    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:11.310330 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:11.310252    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" podStartSLOduration=3.007888452 podStartE2EDuration="3.31023838s" totalImagesPullingTime="302.349928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:11.308550227 +0000 UTC m=+1796.070987132" watchObservedRunningTime="2026-11-02 07:54:11.31023838 +0000 UTC m=+1796.072675286"
Nov 02 07:54:12.113576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.113544    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 07:54:12.146368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.146330    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:12.147465 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.147438    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:12.148501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.148482    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 07:54:12.851909 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.851882    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:54:12.889626 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.889594    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 07:54:12.889790 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.889675    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 07:54:12.962613 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962581    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.962818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962626    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.962818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962662    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.962818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962718    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.962818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962752    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.962818 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962775    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.963075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962862    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:12.963075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.962980    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.963075 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963021    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.963228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963086    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\") pod \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\" (UID: \"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19\") "
Nov 02 07:54:12.963378 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963349    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:12.963913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963871    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:12.963913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963895    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:12.963913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963908    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:12.964145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.963922    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:12.964145 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.964053    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:54:12.969333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.966301    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:12.969333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.966567    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:12.969333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.966672    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:54:12.969333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:12.966704    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz" pod "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" (UID: "260dc4bc-11da-484b-9c1a-6f5d1f2c7a19"). InnerVolumeSpecName "kube-api-access-bhvxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065141    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065170    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065179    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065187    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065197    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065206    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhvxz\" (UniqueName: \"kubernetes.io/projected/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-kube-api-access-bhvxz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.065220 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.065214    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:13.282516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.282483    4515 generic.go:356] "Generic (PLEG): container finished" podID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerID="0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09" exitCode=0
Nov 02 07:54:13.282711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.282532    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"
Nov 02 07:54:13.282711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.282564    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerDied","Data":"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"}
Nov 02 07:54:13.282711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.282620    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt" event={"ID":"260dc4bc-11da-484b-9c1a-6f5d1f2c7a19","Type":"ContainerDied","Data":"0927549de3273690ca1d2f752c0853b654ebaae973c26b5a04811c8ad6fca62f"}
Nov 02 07:54:13.282711 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.282642    4515 scope.go:122] "RemoveContainer" containerID="0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"
Nov 02 07:54:13.293890 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.293866    4515 scope.go:122] "RemoveContainer" containerID="f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"
Nov 02 07:54:13.303656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.303632    4515 scope.go:122] "RemoveContainer" containerID="0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"
Nov 02 07:54:13.303921 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:13.303898    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09\": container with ID starting with 0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09 not found: ID does not exist" containerID="0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"
Nov 02 07:54:13.303963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.303931    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09"} err="failed to get container status \"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09\": rpc error: code = NotFound desc = could not find container \"0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09\": container with ID starting with 0ecd9079ec9b2e7fce9cdd97d012e6395085b4a0934c15e267b1398907689c09 not found: ID does not exist"
Nov 02 07:54:13.303963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.303950    4515 scope.go:122] "RemoveContainer" containerID="f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"
Nov 02 07:54:13.304225 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:13.304206    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e\": container with ID starting with f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e not found: ID does not exist" containerID="f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"
Nov 02 07:54:13.304289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.304230    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e"} err="failed to get container status \"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e\": rpc error: code = NotFound desc = could not find container \"f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e\": container with ID starting with f0cbd351bab808c7219408f13a029db8eb2129c612c9ba4e4cb4141607d4487e not found: ID does not exist"
Nov 02 07:54:13.337211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.337127    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:54:13.351117 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.351088    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-16e21e-openshift-default-66957c786c-t7mzt"]
Nov 02 07:54:13.374196 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:13.374168    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" path="/var/lib/kubelet/pods/260dc4bc-11da-484b-9c1a-6f5d1f2c7a19/volumes"
Nov 02 07:54:15.251364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:15.251300    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 07:54:17.208427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:17.208397    4515 scope.go:122] "RemoveContainer" containerID="c3e10f077ceb3ca1ea5880f4cb551ada92009eb68e02a5651ff978e8c88d0254"
Nov 02 07:54:17.220137 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:17.220101    4515 scope.go:122] "RemoveContainer" containerID="1013cafd25dc505611a6e57add65533491cb628f1245ef54f0d0d81f837a2aee"
Nov 02 07:54:17.279028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:17.279003    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 07:54:17.724033 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:17.723995    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:54:18.111133 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.111092    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:54:18.111856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.111834    4515 cpu_manager.go:414] "containerMap: removing container" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.111982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.111857    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.111982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.111887    4515 cpu_manager.go:414] "containerMap: removing container" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.111982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.111895    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.112237 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.112197    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.112302 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.112248    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="260dc4bc-11da-484b-9c1a-6f5d1f2c7a19" containerName="istio-proxy"
Nov 02 07:54:18.118395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.118373    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.129334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.129297    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:54:18.133734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.133713    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.144115 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.144081    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:54:18.161661 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.160705    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:54:18.260879 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.260847    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.260910    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.260999    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261040    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261096    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261155    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261221    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261347    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261390    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jm8\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261433    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxbm\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261480    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261522    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261569    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261604    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261703    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.261966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261743    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261794    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.261966 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.261861    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.362885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.362784    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.362885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.362831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.362885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.362872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.362901    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.362959    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363003    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363177 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363137    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363211    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363248    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363277    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363294    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363393    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363488 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363444    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363518    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363556    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363586    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363587    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363647    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363657    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363703    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jm8\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363749    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxbm\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363783    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.363862 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363811    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.364429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363899    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.364429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.363896    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.364429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.364027    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.364429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.364278    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.364429 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.364388    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.366222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.366186    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.366356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.366254    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.366356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.366280    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.366356 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.366336    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.387002 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.386976    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.387193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.387174    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.387504 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.387483    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jm8\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.387798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.387781    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxbm\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.433578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.433543    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:18.451571 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.451540    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:18.769753 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:18.769712    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1592d12f_65cb_41d7_8394_42832c403973.slice/crio-cbf02903efd3d4563450b46877a9c681c5c9c5f7acfd6a3518d434ca58f65435 WatchSource:0}: Error finding container cbf02903efd3d4563450b46877a9c681c5c9c5f7acfd6a3518d434ca58f65435: Status 404 returned error can't find the container with id cbf02903efd3d4563450b46877a9c681c5c9c5f7acfd6a3518d434ca58f65435
Nov 02 07:54:18.771597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.771406    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:54:18.772644 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.772609    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.772758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.772681    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.772758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.772713    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.775114 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:18.775092    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a87237_365f_4061_ad26_9865b4accd23.slice/crio-25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a WatchSource:0}: Error finding container 25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a: Status 404 returned error can't find the container with id 25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a
Nov 02 07:54:18.776865 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.776844    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:54:18.778180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.778147    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.778289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.778220    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.778289 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:18.778266    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:19.334188 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.334152    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" event={"ID":"1592d12f-65cb-41d7-8394-42832c403973","Type":"ContainerStarted","Data":"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"}
Nov 02 07:54:19.334685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.334206    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" event={"ID":"1592d12f-65cb-41d7-8394-42832c403973","Type":"ContainerStarted","Data":"cbf02903efd3d4563450b46877a9c681c5c9c5f7acfd6a3518d434ca58f65435"}
Nov 02 07:54:19.335846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.335821    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" event={"ID":"60a87237-365f-4061-ad26-9865b4accd23","Type":"ContainerStarted","Data":"aa450f58cbd1da716c6c99bfddd15d0fb660b3c21ca24c249023c92f7b3f6919"}
Nov 02 07:54:19.335967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.335865    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" event={"ID":"60a87237-365f-4061-ad26-9865b4accd23","Type":"ContainerStarted","Data":"25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a"}
Nov 02 07:54:19.382741 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.382686    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podStartSLOduration=1.382670898 podStartE2EDuration="1.382670898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:19.37939484 +0000 UTC m=+1804.141831747" watchObservedRunningTime="2026-11-02 07:54:19.382670898 +0000 UTC m=+1804.145107889"
Nov 02 07:54:19.427718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.427665    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podStartSLOduration=1.427652613 podStartE2EDuration="1.427652613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:19.42497282 +0000 UTC m=+1804.187409726" watchObservedRunningTime="2026-11-02 07:54:19.427652613 +0000 UTC m=+1804.190089518"
Nov 02 07:54:19.434251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.434225    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:19.436064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.436040    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:19.436169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.436081    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:19.452716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.452684    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:19.454166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.454140    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:19.454283 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:19.454188    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:20.450334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:20.449822    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:20.450334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:20.449904    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:20.464335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:20.454906    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:20.464335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:20.454966    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:21.438334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:21.437710    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:21.438334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:21.437791    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:21.476425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:21.463812    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:21.476425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:21.463871    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:22.444335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:22.441816    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:22.444335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:22.441889    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:22.454824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:22.454789    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:22.455163 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:22.455131    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:23.437332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:23.436718    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:23.437332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:23.436785    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:23.458328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:23.458233    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:23.458328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:23.458296    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:24.436334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:24.434749    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:24.436334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:24.434821    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:24.457273 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:24.457236    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:24.457900 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:24.457864    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:25.339994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:25.339944    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:54:25.491657 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:25.489802    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:25.492913 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:25.492675    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:25.494206 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:25.494021    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:25.497956 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:25.497928    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:26.442374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:26.440708    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 07:54:26.442374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:26.440783    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 07:54:26.458334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:26.455328    4515 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 07:54:26.458334 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:26.455394    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 07:54:27.447652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.447622    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:27.448161 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.448084    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:27.448794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.448776    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:54:27.456318 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.456286    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:27.456607 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.456584    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:27.457213 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.457197    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:54:27.766989 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.766940    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:54:27.767247 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.767197    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" podUID="18324e8c-3844-429c-ae27-605c435ea14b" containerName="maas-api" containerID="cri-o://2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e" gracePeriod=30
Nov 02 07:54:27.959384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.957754    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:54:27.959384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:27.958001    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" podUID="7009650f-2c70-4f4e-bea4-77518b0c5b12" containerName="payload-pre-processing" containerID="cri-o://6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451" gracePeriod=30
Nov 02 07:54:28.178693 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.178661    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:54:28.260333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.253742    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\") pod \"18324e8c-3844-429c-ae27-605c435ea14b\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") "
Nov 02 07:54:28.260333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.253931    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\" (UniqueName: \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\") pod \"18324e8c-3844-429c-ae27-605c435ea14b\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") "
Nov 02 07:54:28.260333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.254009    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") pod \"18324e8c-3844-429c-ae27-605c435ea14b\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") "
Nov 02 07:54:28.260333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.254097    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\") pod \"18324e8c-3844-429c-ae27-605c435ea14b\" (UID: \"18324e8c-3844-429c-ae27-605c435ea14b\") "
Nov 02 07:54:28.299995 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.299939    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca" pod "18324e8c-3844-429c-ae27-605c435ea14b" (UID: "18324e8c-3844-429c-ae27-605c435ea14b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:54:28.378938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.368676    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324e8c-3844-429c-ae27-605c435ea14b-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.436037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.435968    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b" pod "18324e8c-3844-429c-ae27-605c435ea14b" (UID: "18324e8c-3844-429c-ae27-605c435ea14b"). InnerVolumeSpecName "kube-api-access-ddl2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:28.485392 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.473281    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddl2b\" (UniqueName: \"kubernetes.io/projected/18324e8c-3844-429c-ae27-605c435ea14b-kube-api-access-ddl2b\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.489468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.488499    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs" pod "18324e8c-3844-429c-ae27-605c435ea14b" (UID: "18324e8c-3844-429c-ae27-605c435ea14b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:54:28.489468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.488959    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls" pod "18324e8c-3844-429c-ae27-605c435ea14b" (UID: "18324e8c-3844-429c-ae27-605c435ea14b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:54:28.511523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.509921    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:54:28.574781 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.574752    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\" (UniqueName: \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\") pod \"7009650f-2c70-4f4e-bea4-77518b0c5b12\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") "
Nov 02 07:54:28.574981 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.574845    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\") pod \"7009650f-2c70-4f4e-bea4-77518b0c5b12\" (UID: \"7009650f-2c70-4f4e-bea4-77518b0c5b12\") "
Nov 02 07:54:28.575489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.575427    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.575489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.575458    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18324e8c-3844-429c-ae27-605c435ea14b-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.575678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.575630    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume" pod "7009650f-2c70-4f4e-bea4-77518b0c5b12" (UID: "7009650f-2c70-4f4e-bea4-77518b0c5b12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:54:28.577368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.577339    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw" pod "7009650f-2c70-4f4e-bea4-77518b0c5b12" (UID: "7009650f-2c70-4f4e-bea4-77518b0c5b12"). InnerVolumeSpecName "kube-api-access-7d8fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:28.604025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.603938    4515 generic.go:356] "Generic (PLEG): container finished" podID="18324e8c-3844-429c-ae27-605c435ea14b" containerID="2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e" exitCode=0
Nov 02 07:54:28.604025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.603988    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"
Nov 02 07:54:28.604219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.604029    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" event={"ID":"18324e8c-3844-429c-ae27-605c435ea14b","Type":"ContainerDied","Data":"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"}
Nov 02 07:54:28.604219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.604103    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9" event={"ID":"18324e8c-3844-429c-ae27-605c435ea14b","Type":"ContainerDied","Data":"8b436d67a00c878fbf281e17794f15b0d5187921a74739c046afce717f382445"}
Nov 02 07:54:28.604219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.604140    4515 scope.go:122] "RemoveContainer" containerID="2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"
Nov 02 07:54:28.606091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.606065    4515 generic.go:356] "Generic (PLEG): container finished" podID="7009650f-2c70-4f4e-bea4-77518b0c5b12" containerID="6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451" exitCode=0
Nov 02 07:54:28.606221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.606104    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"
Nov 02 07:54:28.606221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.606145    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" event={"ID":"7009650f-2c70-4f4e-bea4-77518b0c5b12","Type":"ContainerDied","Data":"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"}
Nov 02 07:54:28.606221 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.606188    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw" event={"ID":"7009650f-2c70-4f4e-bea4-77518b0c5b12","Type":"ContainerDied","Data":"da036b7ffa49568c694eeff2bfc677e0aa0b4e517bb1cdc68d91fd41b81387ae"}
Nov 02 07:54:28.653212 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.653172    4515 scope.go:122] "RemoveContainer" containerID="2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"
Nov 02 07:54:28.653618 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:28.653595    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e\": container with ID starting with 2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e not found: ID does not exist" containerID="2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"
Nov 02 07:54:28.653686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.653629    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e"} err="failed to get container status \"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e\": rpc error: code = NotFound desc = could not find container \"2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e\": container with ID starting with 2caf4f8e8ac1d998781c23de86f0eebab49587fd48105b4008a23463f3729e7e not found: ID does not exist"
Nov 02 07:54:28.653686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.653648    4515 scope.go:122] "RemoveContainer" containerID="6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"
Nov 02 07:54:28.664070 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.664048    4515 scope.go:122] "RemoveContainer" containerID="6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"
Nov 02 07:54:28.664560 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:28.664525    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451\": container with ID starting with 6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451 not found: ID does not exist" containerID="6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"
Nov 02 07:54:28.664646 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.664580    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451"} err="failed to get container status \"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451\": rpc error: code = NotFound desc = could not find container \"6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451\": container with ID starting with 6cbe39730d173ec647995b952440c96b17e97bf5341e618022bb53e89fa9d451 not found: ID does not exist"
Nov 02 07:54:28.677727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.677700    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d8fw\" (UniqueName: \"kubernetes.io/projected/7009650f-2c70-4f4e-bea4-77518b0c5b12-kube-api-access-7d8fw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.677727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.677731    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7009650f-2c70-4f4e-bea4-77518b0c5b12-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.690069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.690041    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:54:28.694266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.694245    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-677726-686c545f7c-qf4t9"]
Nov 02 07:54:28.745580 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.745545    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:54:28.748830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:28.748800    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-766f5b99ff-c5hbw"]
Nov 02 07:54:29.374179 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:29.374141    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18324e8c-3844-429c-ae27-605c435ea14b" path="/var/lib/kubelet/pods/18324e8c-3844-429c-ae27-605c435ea14b/volumes"
Nov 02 07:54:29.374679 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:29.374662    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7009650f-2c70-4f4e-bea4-77518b0c5b12" path="/var/lib/kubelet/pods/7009650f-2c70-4f4e-bea4-77518b0c5b12/volumes"
Nov 02 07:54:31.176343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:31.174817    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:54:31.176343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:31.175085    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy" containerID="cri-o://6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1" gracePeriod=30
Nov 02 07:54:36.607224 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.607194    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:54:36.665716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.665667    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.665924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.665841    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.665924 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.665894    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.666073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.665942    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.666073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.665980    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.667055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.667021    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:36.667191 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.666011    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.667259 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.667215    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.667333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.667286    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.667401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.667351    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\") pod \"f2204be0-ff01-4619-acd9-c5b2795c14ed\" (UID: \"f2204be0-ff01-4619-acd9-c5b2795c14ed\") "
Nov 02 07:54:36.667978 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.667953    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.668808 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.668781    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:36.669931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.669447    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:36.669931 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.669606    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:54:36.670430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.670399    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:36.694962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.694631    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:54:36.696703 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.696675    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:54:36.697456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697433    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697457    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697504    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697513    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697530    4515 cpu_manager.go:414] "containerMap: removing container" podUID="18324e8c-3844-429c-ae27-605c435ea14b" containerName="maas-api"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697537    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18324e8c-3844-429c-ae27-605c435ea14b" containerName="maas-api"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697574    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7009650f-2c70-4f4e-bea4-77518b0c5b12" containerName="payload-pre-processing"
Nov 02 07:54:36.697623 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697582    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7009650f-2c70-4f4e-bea4-77518b0c5b12" containerName="payload-pre-processing"
Nov 02 07:54:36.697963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697719    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697741    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerName="istio-proxy"
Nov 02 07:54:36.697963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697749    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="18324e8c-3844-429c-ae27-605c435ea14b" containerName="maas-api"
Nov 02 07:54:36.697963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.697758    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7009650f-2c70-4f4e-bea4-77518b0c5b12" containerName="payload-pre-processing"
Nov 02 07:54:36.702084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.701478    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:54:36.706214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.702451    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "kube-api-access-jqpkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:36.706214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.702631    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.706214 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.703914    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token" pod "f2204be0-ff01-4619-acd9-c5b2795c14ed" (UID: "f2204be0-ff01-4619-acd9-c5b2795c14ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:36.720328 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.717944    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:54:36.736368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.720645    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"default-dockercfg-858ql\""
Nov 02 07:54:36.736368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.720793    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"openshift-service-ca.crt\""
Nov 02 07:54:36.736368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.720874    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"kube-root-ca.crt\""
Nov 02 07:54:36.736368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.720985    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"auth-test-model-9877ce-kserve-self-signed-certs\""
Nov 02 07:54:36.774410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.768873    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.768933    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774410 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774361    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcq4q\" (UniqueName: \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774476    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774619    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774794    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqpkm\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-kube-api-access-jqpkm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774813    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774830    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774846    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2204be0-ff01-4619-acd9-c5b2795c14ed-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774860    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774873    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774887    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2204be0-ff01-4619-acd9-c5b2795c14ed-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.774916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.774899    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2204be0-ff01-4619-acd9-c5b2795c14ed-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:54:36.780009 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.779975    4515 generic.go:356] "Generic (PLEG): container finished" podID="f2204be0-ff01-4619-acd9-c5b2795c14ed" containerID="6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1" exitCode=0
Nov 02 07:54:36.780164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.780012    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerDied","Data":"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"}
Nov 02 07:54:36.780164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.780094    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx" event={"ID":"f2204be0-ff01-4619-acd9-c5b2795c14ed","Type":"ContainerDied","Data":"ef54692b30411d3c2e12a49424ae9f42cd25b083eb58f43b3e8816be385db065"}
Nov 02 07:54:36.780164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.780108    4515 scope.go:122] "RemoveContainer" containerID="6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"
Nov 02 07:54:36.780373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.780207    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"
Nov 02 07:54:36.802011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.799346    4515 scope.go:122] "RemoveContainer" containerID="8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"
Nov 02 07:54:36.850600 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.850575    4515 scope.go:122] "RemoveContainer" containerID="6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"
Nov 02 07:54:36.851046 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:36.851022    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1\": container with ID starting with 6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1 not found: ID does not exist" containerID="6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"
Nov 02 07:54:36.851183 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.851065    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1"} err="failed to get container status \"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1\": rpc error: code = NotFound desc = could not find container \"6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1\": container with ID starting with 6904ec47b7fc322760008c4e386811d6eb4e1c6a7dfb6c2ce90ea511bb1b9eb1 not found: ID does not exist"
Nov 02 07:54:36.851258 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.851204    4515 scope.go:122] "RemoveContainer" containerID="8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"
Nov 02 07:54:36.851653 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:36.851627    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a\": container with ID starting with 8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a not found: ID does not exist" containerID="8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"
Nov 02 07:54:36.851728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.851663    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a"} err="failed to get container status \"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a\": rpc error: code = NotFound desc = could not find container \"8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a\": container with ID starting with 8f5c9fedeac8dee739ac475493b7db1d17e8ef55f2dcf8c433ad46242ace585a not found: ID does not exist"
Nov 02 07:54:36.853699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.853674    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:54:36.857902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.857876    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-677726-openshift-default-6c9cff679f-tfmfx"]
Nov 02 07:54:36.876614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876583    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.876614 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876627    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876698    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876720    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcq4q\" (UniqueName: \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876747    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877084 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.876829    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877373 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.877195    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877764 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.877745    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.877897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.877854    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.879897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.879874    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.880097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.880080    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:36.901880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:36.901850    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcq4q\" (UniqueName: \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\") pod \"auth-test-model-9877ce-kserve-6d766b4f57-zhhh8\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:37.074955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.074577    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:37.381664 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.381584    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2204be0-ff01-4619-acd9-c5b2795c14ed" path="/var/lib/kubelet/pods/f2204be0-ff01-4619-acd9-c5b2795c14ed/volumes"
Nov 02 07:54:37.551028 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.550986    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:54:37.556514 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:37.556481    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf191ed8f_210e_46ea_be8d_3cdf2d6a0629.slice/crio-d7d1ca7998739bc6ae9bd502b042f2466944a7a0aba85a9b3c0f806356754af8 WatchSource:0}: Error finding container d7d1ca7998739bc6ae9bd502b042f2466944a7a0aba85a9b3c0f806356754af8: Status 404 returned error can't find the container with id d7d1ca7998739bc6ae9bd502b042f2466944a7a0aba85a9b3c0f806356754af8
Nov 02 07:54:37.790727 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.790685    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" event={"ID":"f191ed8f-210e-46ea-be8d-3cdf2d6a0629","Type":"ContainerStarted","Data":"472e46d51b6756570235608932e5c8cef991827a34386c46c2b8f2ad2b89a509"}
Nov 02 07:54:37.791223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.790738    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" event={"ID":"f191ed8f-210e-46ea-be8d-3cdf2d6a0629","Type":"ContainerStarted","Data":"d7d1ca7998739bc6ae9bd502b042f2466944a7a0aba85a9b3c0f806356754af8"}
Nov 02 07:54:37.847193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:37.847121    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" podStartSLOduration=1.847100464 podStartE2EDuration="1.847100464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:37.844701161 +0000 UTC m=+1822.607138067" watchObservedRunningTime="2026-11-02 07:54:37.847100464 +0000 UTC m=+1822.609537374"
Nov 02 07:54:38.075831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.075715    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:38.800250 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.800216    4515 generic.go:356] "Generic (PLEG): container finished" podID="159a0cec-6643-43c9-9e64-33a38e868887" containerID="1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a" exitCode=0
Nov 02 07:54:38.800675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.800286    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerDied","Data":"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"}
Nov 02 07:54:38.800819 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.800799    4515 scope.go:122] "RemoveContainer" containerID="1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"
Nov 02 07:54:38.802401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.802368    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:38.802556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.802455    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:38.802556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:38.802503    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:39.809558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:39.809518    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerStarted","Data":"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"}
Nov 02 07:54:40.100596 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.085446    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:54:40.130897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.117153    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:54:40.130897 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.117647    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.186331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.182971    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c7a27f\""
Nov 02 07:54:40.186331 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.183144    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c7a27f-dockercfg-xrwf7\""
Nov 02 07:54:40.271360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.271302    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jklc\" (UniqueName: \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.271564 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.271387    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.375234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.375129    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jklc\" (UniqueName: \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.375234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.375184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.376175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.376148    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.384992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.384969    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:54:40.390260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.390231    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:40.390394 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.390278    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:40.399138 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.399116    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jklc\" (UniqueName: \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\") pod \"payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.496901 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.496860    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:40.978200 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:40.978169    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7528a6f_530c_43cc_88b3_f622254b43d5.slice/crio-b4fd162a6f72a80b175bac1591864e89ff8b87af8f0275c046df3817c75f66c3 WatchSource:0}: Error finding container b4fd162a6f72a80b175bac1591864e89ff8b87af8f0275c046df3817c75f66c3: Status 404 returned error can't find the container with id b4fd162a6f72a80b175bac1591864e89ff8b87af8f0275c046df3817c75f66c3
Nov 02 07:54:40.979011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:40.978986    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:54:41.384266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:41.384230    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:41.384491 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:41.384275    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:41.947902 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:41.947867    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" event={"ID":"c7528a6f-530c-43cc-88b3-f622254b43d5","Type":"ContainerStarted","Data":"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"}
Nov 02 07:54:41.948057 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:41.947922    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" event={"ID":"c7528a6f-530c-43cc-88b3-f622254b43d5","Type":"ContainerStarted","Data":"b4fd162a6f72a80b175bac1591864e89ff8b87af8f0275c046df3817c75f66c3"}
Nov 02 07:54:41.948146 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:41.948121    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:42.000466 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:42.000405    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" podStartSLOduration=3.000385365 podStartE2EDuration="3.000385365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:41.995328157 +0000 UTC m=+1826.757765066" watchObservedRunningTime="2026-11-02 07:54:42.000385365 +0000 UTC m=+1826.762822275"
Nov 02 07:54:42.387033 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:42.386996    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:42.387228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:42.387048    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:42.557164 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:42.557128    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:54:43.375835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.375799    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 07:54:43.381045 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.381022    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.385013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.384986    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:43.385158 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.385027    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:43.391884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.391857    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 07:54:43.400074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.400048    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3575b9\""
Nov 02 07:54:43.400249 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.400191    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3575b9-dockercfg-4c558\""
Nov 02 07:54:43.512832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.512796    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.522532 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.522479    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsn9g\" (UniqueName: \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.692531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.690686    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.692531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.690771    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsn9g\" (UniqueName: \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.692531 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.692294    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.751516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.751032    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsn9g\" (UniqueName: \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\") pod \"payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:43.999201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:43.999159    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:44.392957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.392443    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:44.392957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.392510    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:44.554793 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.554699    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 07:54:44.613464 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.613425    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:54:44.613620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.613491    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:54:44.977656 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.977561    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" event={"ID":"1fecf749-d5c7-44ab-8831-b79bf840d334","Type":"ContainerStarted","Data":"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"}
Nov 02 07:54:44.980782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.980744    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:44.980964 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:44.980795    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" event={"ID":"1fecf749-d5c7-44ab-8831-b79bf840d334","Type":"ContainerStarted","Data":"b737f94114b654f4d7a223662bec6eb19ff35c573d8cd45fd70883976fc1f305"}
Nov 02 07:54:45.033180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:45.033117    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" podStartSLOduration=2.033097554 podStartE2EDuration="2.033097554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:45.032703004 +0000 UTC m=+1829.795139941" watchObservedRunningTime="2026-11-02 07:54:45.033097554 +0000 UTC m=+1829.795534460"
Nov 02 07:54:45.384934 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:45.384895    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 02 07:54:45.385555 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:45.385299    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 02 07:54:46.390368 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:46.390303    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:54:46.391419 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:46.391391    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 07:54:47.075749 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:47.075715    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:47.093516 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:47.093453    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:47.745663 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:54:47.745615    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:54:48.003109 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.003073    4515 generic.go:356] "Generic (PLEG): container finished" podID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerID="548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b" exitCode=0
Nov 02 07:54:48.003290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.003145    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerDied","Data":"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"}
Nov 02 07:54:48.003803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.003778    4515 scope.go:122] "RemoveContainer" containerID="548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"
Nov 02 07:54:48.005352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.005292    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:48.005443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.005395    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:48.005443 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.005438    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:48.046856 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:48.046824    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:54:49.013270 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:49.013231    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerStarted","Data":"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"}
Nov 02 07:54:49.077549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:49.077512    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:54:49.077736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:49.077686    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:49.077814 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:49.077746    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:50.077847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:50.077813    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:50.078346 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:50.078079    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:51.078119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:51.078081    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:51.078675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:51.078155    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:52.077898 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:52.077859    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:52.078119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:52.077934    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:52.959831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:52.959800    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:54:53.077421 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:53.077388    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:53.077594 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:53.077455    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:54.077519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:54.077483    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:54.077958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:54.077553    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:55.077836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:55.077788    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:54:55.078269 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:55.077873    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 07:54:55.989498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:55.989465    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 07:54:56.084507 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:56.084477    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:54:56.085023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:56.084991    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:54:56.085852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:56.085830    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 07:54:56.438684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:56.436884    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": read tcp 10.133.0.2:46604->10.133.0.188:15021: read: connection reset by peer" start-of-body=
Nov 02 07:54:56.438684 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:56.436960    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": read tcp 10.133.0.2:46604->10.133.0.188:15021: read: connection reset by peer"
Nov 02 07:54:57.124686 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.124583    4515 generic.go:356] "Generic (PLEG): container finished" podID="03afe710-f612-4504-9c63-e282b67993f8" containerID="1efd6b2fdf359b1dc9f1372b7f1d87a67d3c80e65bb2a9a7610bd1b4c4802d7c" exitCode=0
Nov 02 07:54:57.125426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.124646    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerDied","Data":"1efd6b2fdf359b1dc9f1372b7f1d87a67d3c80e65bb2a9a7610bd1b4c4802d7c"}
Nov 02 07:54:57.126210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.126185    4515 scope.go:122] "RemoveContainer" containerID="1efd6b2fdf359b1dc9f1372b7f1d87a67d3c80e65bb2a9a7610bd1b4c4802d7c"
Nov 02 07:54:57.128828 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.128787    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:57.129210 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.129180    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:57.130015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:57.129243    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:58.149031 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.148930    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerStarted","Data":"b99fd3f0dca6d612d2b4a5b860cb583220999c7413b1ced03e0d676a158ef3ce"}
Nov 02 07:54:58.281108 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.281073    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:54:58.281343 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.281284    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:54:58.281407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.281384    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:54:58.715687 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.715645    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:54:58.757803 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.757756    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:54:58.758016 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.757889    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.777439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.777409    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"default-dockercfg-k7sph\""
Nov 02 07:54:58.777439 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.777444    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"auth-test-model-12df3c-kserve-self-signed-certs\""
Nov 02 07:54:58.779899 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.779814    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"openshift-service-ca.crt\""
Nov 02 07:54:58.789720 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.789701    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"kube-root-ca.crt\""
Nov 02 07:54:58.815236 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815210    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.815395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815338    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.815395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815366    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.815523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815502    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.815599 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815587    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.815685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.815672    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbh8m\" (UniqueName: \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.933470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.933421    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.941587 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.941550    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.942422 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.942396    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.943231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.943210    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.954332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.943851    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.955769 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.942177    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.956068 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.934655    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.956204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.944643    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.956426 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.955053    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.956992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.956970    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh8m\" (UniqueName: \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:58.973266 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:58.973170    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:59.021533 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:59.021506    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh8m\" (UniqueName: \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\") pod \"auth-test-model-12df3c-kserve-b567bcdcc-w46jn\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:59.079659 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:59.079621    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:54:59.281119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:59.280683    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:54:59.281119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:59.280740    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:54:59.497719 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:54:59.497649    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc466660d_76e7_4878_9ba5_5063c956691b.slice/crio-9c6a670c7f62f6b2a013b18a884a700e2bb972022f64b498212fa65bce1b10b2 WatchSource:0}: Error finding container 9c6a670c7f62f6b2a013b18a884a700e2bb972022f64b498212fa65bce1b10b2: Status 404 returned error can't find the container with id 9c6a670c7f62f6b2a013b18a884a700e2bb972022f64b498212fa65bce1b10b2
Nov 02 07:54:59.498994 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:54:59.498969    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:55:00.184695 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:00.184649    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" event={"ID":"c466660d-76e7-4878-9ba5-5063c956691b","Type":"ContainerStarted","Data":"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"}
Nov 02 07:55:00.184695 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:00.184701    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" event={"ID":"c466660d-76e7-4878-9ba5-5063c956691b","Type":"ContainerStarted","Data":"9c6a670c7f62f6b2a013b18a884a700e2bb972022f64b498212fa65bce1b10b2"}
Nov 02 07:55:00.281372 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:00.281336    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:55:00.281857 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:00.281397    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:55:01.084758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:01.084716    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:55:01.281395 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:01.281360    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:55:01.281802 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:01.281415    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:55:02.153060 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.152992    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" podStartSLOduration=4.152976906 podStartE2EDuration="4.152976906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:00.244573478 +0000 UTC m=+1845.007010386" watchObservedRunningTime="2026-11-02 07:55:02.152976906 +0000 UTC m=+1846.915413812"
Nov 02 07:55:02.153490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.153470    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 07:55:02.157962 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.157940    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.183095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.183052    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 07:55:02.187193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.187160    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-6929a4\""
Nov 02 07:55:02.187671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.187642    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-6929a4-dockercfg-nfqzt\""
Nov 02 07:55:02.253364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.253280    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x42\" (UniqueName: \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.253582 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.253544    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.254155 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.254131    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 07:55:02.258556 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.258535    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.281199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.281163    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:55:02.281513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.281480    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:55:02.283026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.283000    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 07:55:02.412407 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.389220    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.413255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.413217    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.417597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.417560    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x42\" (UniqueName: \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.440923 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.440884    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrv6\" (UniqueName: \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.441370 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.414708    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.504114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.504074    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x42\" (UniqueName: \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\") pod \"payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.543333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.543271    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.543928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.543898    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrv6\" (UniqueName: \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.567333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.545430    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.622333 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.613893    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrv6\" (UniqueName: \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\") pod \"payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:02.787573 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.787538    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:02.879809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:02.878043    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:03.214357 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:03.214210    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 07:55:03.281277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:03.281217    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 07:55:03.281277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:03.281271    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 07:55:03.296011 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:03.295981    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 07:55:03.310944 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:55:03.310905    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5402fa8e_57da_45a6_a603_bb2da47ba2d6.slice/crio-8c0ee934b05f2d68d21c489519c94911f196d80d0ecadf8902674233f14d8fc4 WatchSource:0}: Error finding container 8c0ee934b05f2d68d21c489519c94911f196d80d0ecadf8902674233f14d8fc4: Status 404 returned error can't find the container with id 8c0ee934b05f2d68d21c489519c94911f196d80d0ecadf8902674233f14d8fc4
Nov 02 07:55:04.261024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.260973    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" event={"ID":"492d1166-1eb2-4523-a33f-80045ef3b210","Type":"ContainerStarted","Data":"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"}
Nov 02 07:55:04.261241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.261044    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" event={"ID":"492d1166-1eb2-4523-a33f-80045ef3b210","Type":"ContainerStarted","Data":"7c4ed5a5fc60610725f8b2084011dd5578e147237863dc64ac8b2b00d617a579"}
Nov 02 07:55:04.261241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.261078    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:04.262920 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.262888    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" event={"ID":"5402fa8e-57da-45a6-a603-bb2da47ba2d6","Type":"ContainerStarted","Data":"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"}
Nov 02 07:55:04.263046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.262941    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" event={"ID":"5402fa8e-57da-45a6-a603-bb2da47ba2d6","Type":"ContainerStarted","Data":"8c0ee934b05f2d68d21c489519c94911f196d80d0ecadf8902674233f14d8fc4"}
Nov 02 07:55:04.263046 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.262972    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:04.285605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.285576    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:55:04.285987 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.285963    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:55:04.286554 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.286533    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 07:55:04.305490 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.305438    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" podStartSLOduration=2.305424105 podStartE2EDuration="2.305424105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:04.303666168 +0000 UTC m=+1849.066103075" watchObservedRunningTime="2026-11-02 07:55:04.305424105 +0000 UTC m=+1849.067861011"
Nov 02 07:55:04.346434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:04.346382    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" podStartSLOduration=2.346368184 podStartE2EDuration="2.346368184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:04.345392024 +0000 UTC m=+1849.107828930" watchObservedRunningTime="2026-11-02 07:55:04.346368184 +0000 UTC m=+1849.108805090"
Nov 02 07:55:08.836086 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.836049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:55:08.950926 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.950888    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:55:08.955743 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.955720    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:08.972627 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.972597    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"openshift-service-ca.crt\""
Nov 02 07:55:08.972767 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.972639    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"kube-root-ca.crt\""
Nov 02 07:55:08.974156 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.974139    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:55:08.982717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.982692    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"auth-test-model-7f19e8-kserve-self-signed-certs\""
Nov 02 07:55:08.982717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:08.982694    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"default-dockercfg-9b42f\""
Nov 02 07:55:09.081377 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.081347    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:55:09.095933 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.095853    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:55:09.122218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122181    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.122414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122331    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.122414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122365    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4jz\" (UniqueName: \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.122414 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122404    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.122605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122462    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.122605 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.122487    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230335    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230421    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4jz\" (UniqueName: \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230490    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230616    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.230723    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.231262    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.231597    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.235340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.232583    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.260332 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.260052    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.276335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.274599    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4jz\" (UniqueName: \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.276335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.275917    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\") pod \"auth-test-model-7f19e8-kserve-79945cfcc-4bmbt\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:09.329643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.329606    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:55:09.579912 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:09.576408    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:10.091340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:10.085173    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:55:10.116368 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:55:10.107882    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21206eb1_2807_4f59_9936_65c47bb9ea56.slice/crio-aa31ee4434494aee4c7cb2239b6b86415f43dfc9baabddf2087853b0430ae42a WatchSource:0}: Error finding container aa31ee4434494aee4c7cb2239b6b86415f43dfc9baabddf2087853b0430ae42a: Status 404 returned error can't find the container with id aa31ee4434494aee4c7cb2239b6b86415f43dfc9baabddf2087853b0430ae42a
Nov 02 07:55:10.315523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:10.315486    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" event={"ID":"21206eb1-2807-4f59-9936-65c47bb9ea56","Type":"ContainerStarted","Data":"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"}
Nov 02 07:55:10.315801 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:10.315774    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" event={"ID":"21206eb1-2807-4f59-9936-65c47bb9ea56","Type":"ContainerStarted","Data":"aa31ee4434494aee4c7cb2239b6b86415f43dfc9baabddf2087853b0430ae42a"}
Nov 02 07:55:10.360492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:10.360223    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" podStartSLOduration=2.360203027 podStartE2EDuration="2.360203027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:10.356816662 +0000 UTC m=+1855.119253583" watchObservedRunningTime="2026-11-02 07:55:10.360203027 +0000 UTC m=+1855.122639934"
Nov 02 07:55:10.580560 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:10.580509    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:15.272992 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:15.272961    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 07:55:15.273536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:15.273047    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 07:55:17.311525 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:17.311495    4515 scope.go:122] "RemoveContainer" containerID="808c39ccb8fab7e9c86b7c0b429f2322dd86339cfcfb699b3ed6a79199e110e1"
Nov 02 07:55:17.724007 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:55:17.723919    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:55:19.577579 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:19.577546    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:19.591277 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:19.591248    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:20.414192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:20.414161    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:55:31.264112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.264068    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:55:31.271523 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.271493    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.291036 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.291008    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:55:31.298527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.298293    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"openshift-service-ca.crt\""
Nov 02 07:55:31.298527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.298475    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"kube-root-ca.crt\""
Nov 02 07:55:31.299874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.299747    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"default-dockercfg-j4wmw\""
Nov 02 07:55:31.300127 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.299923    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"auth-test-model-56bbb6-kserve-self-signed-certs\""
Nov 02 07:55:31.349401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349354    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.349401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.349620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349529    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.349739 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349715    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w7b\" (UniqueName: \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.349806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349792    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.349908 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.349895    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451265 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451221    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451473 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451434    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w7b\" (UniqueName: \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451519 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451503    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451643 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451627    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451685 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451665    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451737 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451694    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.451804 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451741    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.452024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.451994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.452024 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.452009    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.453836 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.453813    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.454239 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.454216    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.475240 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.475207    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w7b\" (UniqueName: \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\") pod \"auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:31.598409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:31.587593    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:32.148445 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:55:32.148412    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod854e735d_24d2_4025_9a01_2195b2dfab4e.slice/crio-bd35f622ed07ed7ee3a3de194f8012beab998fcaadd0798308868a78b09f8e3d WatchSource:0}: Error finding container bd35f622ed07ed7ee3a3de194f8012beab998fcaadd0798308868a78b09f8e3d: Status 404 returned error can't find the container with id bd35f622ed07ed7ee3a3de194f8012beab998fcaadd0798308868a78b09f8e3d
Nov 02 07:55:32.150056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:32.150023    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:55:32.656382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:32.653521    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" event={"ID":"854e735d-24d2-4025-9a01-2195b2dfab4e","Type":"ContainerStarted","Data":"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"}
Nov 02 07:55:32.656382 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:32.653596    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" event={"ID":"854e735d-24d2-4025-9a01-2195b2dfab4e","Type":"ContainerStarted","Data":"bd35f622ed07ed7ee3a3de194f8012beab998fcaadd0798308868a78b09f8e3d"}
Nov 02 07:55:32.699852 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:32.699802    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" podStartSLOduration=1.69978919 podStartE2EDuration="1.69978919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:32.697024113 +0000 UTC m=+1877.459461038" watchObservedRunningTime="2026-11-02 07:55:32.69978919 +0000 UTC m=+1877.462226097"
Nov 02 07:55:33.588058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:33.588019    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:41.588171 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:41.588133    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:41.600845 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:41.600821    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:41.794588 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:41.794550    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:55:47.723775 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:55:47.723738    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:55:52.483423 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:55:52.483384    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:56:08.537166 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.537126    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 07:56:08.544374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.544344    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.558401 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.558373    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 07:56:08.566716 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.566675    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e4c131\"/\"openshift-service-ca.crt\""
Nov 02 07:56:08.572616 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.572588    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e4c131\"/\"auth-test-model-e4c131-kserve-self-signed-certs\""
Nov 02 07:56:08.572805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.572617    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e4c131\"/\"default-dockercfg-lgwxc\""
Nov 02 07:56:08.572805 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.572702    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e4c131\"/\"kube-root-ca.crt\""
Nov 02 07:56:08.594228 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594190    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.594436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5rp\" (UniqueName: \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.594505 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594458    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.594546 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594527    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.594705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594678    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.594839 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.594787    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696041    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5rp\" (UniqueName: \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696087 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696092    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696264    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696435 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696401    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696457    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696566 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696547    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.696880 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696856    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.697013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.696937    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.697235 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.697211    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.699241 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.699218    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.699445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.699425    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.722481 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.722447    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5rp\" (UniqueName: \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\") pod \"auth-test-model-e4c131-kserve-787d958df7-vrzsh\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") " pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:08.868385 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:08.864604    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:09.234016 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:56:09.233986    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f162cd_3ba1_43a6_ad53_622c47e7e392.slice/crio-c105b318ed7d5d5d375e5dbcc620a17e14ef504703289ec7ac6215c6ffd588a0 WatchSource:0}: Error finding container c105b318ed7d5d5d375e5dbcc620a17e14ef504703289ec7ac6215c6ffd588a0: Status 404 returned error can't find the container with id c105b318ed7d5d5d375e5dbcc620a17e14ef504703289ec7ac6215c6ffd588a0
Nov 02 07:56:09.236495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:09.236466    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 07:56:10.056067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:10.056023    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" event={"ID":"e5f162cd-3ba1-43a6-ad53-622c47e7e392","Type":"ContainerStarted","Data":"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"}
Nov 02 07:56:10.056551 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:10.056079    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" event={"ID":"e5f162cd-3ba1-43a6-ad53-622c47e7e392","Type":"ContainerStarted","Data":"c105b318ed7d5d5d375e5dbcc620a17e14ef504703289ec7ac6215c6ffd588a0"}
Nov 02 07:56:10.101482 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:10.101414    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" podStartSLOduration=2.101395076 podStartE2EDuration="2.101395076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:56:10.100023027 +0000 UTC m=+1914.862459934" watchObservedRunningTime="2026-11-02 07:56:10.101395076 +0000 UTC m=+1914.863831984"
Nov 02 07:56:10.866264 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:10.866228    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:17.731919 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:56:17.731866    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:56:18.865806 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:18.865769    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:18.878807 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:18.878779    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:19.132424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:19.132345    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 07:56:28.089387 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:28.085205    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:56:47.724970 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:56:47.724924    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:56:48.014539 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:48.014490    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:56:53.015456 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.015409    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" podUID="c7528a6f-530c-43cc-88b3-f622254b43d5" containerName="payload-processing" containerID="cri-o://f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5" gracePeriod=25
Nov 02 07:56:53.298911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.298880    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:56:53.374095 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.374052    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\" (UniqueName: \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\") pod \"c7528a6f-530c-43cc-88b3-f622254b43d5\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") "
Nov 02 07:56:53.374352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.374325    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\") pod \"c7528a6f-530c-43cc-88b3-f622254b43d5\" (UID: \"c7528a6f-530c-43cc-88b3-f622254b43d5\") "
Nov 02 07:56:53.374938 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.374904    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume" pod "c7528a6f-530c-43cc-88b3-f622254b43d5" (UID: "c7528a6f-530c-43cc-88b3-f622254b43d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:56:53.376578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.376548    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc" pod "c7528a6f-530c-43cc-88b3-f622254b43d5" (UID: "c7528a6f-530c-43cc-88b3-f622254b43d5"). InnerVolumeSpecName "kube-api-access-9jklc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:56:53.388143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.387832    4515 generic.go:356] "Generic (PLEG): container finished" podID="c7528a6f-530c-43cc-88b3-f622254b43d5" containerID="f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5" exitCode=0
Nov 02 07:56:53.388143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.387880    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" event={"ID":"c7528a6f-530c-43cc-88b3-f622254b43d5","Type":"ContainerDied","Data":"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"}
Nov 02 07:56:53.388143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.387928    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7" event={"ID":"c7528a6f-530c-43cc-88b3-f622254b43d5","Type":"ContainerDied","Data":"b4fd162a6f72a80b175bac1591864e89ff8b87af8f0275c046df3817c75f66c3"}
Nov 02 07:56:53.388143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.387951    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"
Nov 02 07:56:53.388143 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.387959    4515 scope.go:122] "RemoveContainer" containerID="f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"
Nov 02 07:56:53.407841 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.407816    4515 scope.go:122] "RemoveContainer" containerID="f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"
Nov 02 07:56:53.408208 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:56:53.408179    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5\": container with ID starting with f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5 not found: ID does not exist" containerID="f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"
Nov 02 07:56:53.408329 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.408218    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5"} err="failed to get container status \"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5\": rpc error: code = NotFound desc = could not find container \"f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5\": container with ID starting with f4592926e4a3f31b18317e58f796784d854d17128e57b1bf62b92fb024fd74d5 not found: ID does not exist"
Nov 02 07:56:53.443654 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.443624    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:56:53.447810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.447786    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7a27f-85d5f4c795-62hq7"]
Nov 02 07:56:53.476384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.476348    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jklc\" (UniqueName: \"kubernetes.io/projected/c7528a6f-530c-43cc-88b3-f622254b43d5-kube-api-access-9jklc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:56:53.476384 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:53.476382    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7528a6f-530c-43cc-88b3-f622254b43d5-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:56:55.375204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:56:55.375168    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7528a6f-530c-43cc-88b3-f622254b43d5" path="/var/lib/kubelet/pods/c7528a6f-530c-43cc-88b3-f622254b43d5/volumes"
Nov 02 07:57:08.343360 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:57:08.343325    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:57:17.728638 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:57:17.728551    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:57:40.293732 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:57:40.293695    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:57:47.723772 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:57:47.723733    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:58:09.837211 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:09.837129    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:58:09.837704 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:09.837389    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy" containerID="cri-o://fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2" gracePeriod=30
Nov 02 07:58:15.139527 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.139500    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:58:15.201449 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201347    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201451    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201489    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201627    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201674    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201767    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:15.201810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201790    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.201957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201862    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.202020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.201966    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.202020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202009    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\") pod \"12f7a25a-f059-4817-a209-e91edf3115b0\" (UID: \"12f7a25a-f059-4817-a209-e91edf3115b0\") "
Nov 02 07:58:15.202135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202070    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:15.202135 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202070    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:15.202402 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202376    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:15.202756 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202730    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.202877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202759    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.202877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202772    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.202877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.202787    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f7a25a-f059-4817-a209-e91edf3115b0-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.204386 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.204362    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:15.204549 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.204534    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:15.204809 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.204794    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "kube-api-access-gwq4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:15.204939 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.204913    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:15.209431 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.209402    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket" pod "12f7a25a-f059-4817-a209-e91edf3115b0" (UID: "12f7a25a-f059-4817-a209-e91edf3115b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:15.233262 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.233233    4515 generic.go:356] "Generic (PLEG): container finished" podID="12f7a25a-f059-4817-a209-e91edf3115b0" containerID="fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2" exitCode=0
Nov 02 07:58:15.233416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.233287    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"
Nov 02 07:58:15.233416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.233337    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" event={"ID":"12f7a25a-f059-4817-a209-e91edf3115b0","Type":"ContainerDied","Data":"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"}
Nov 02 07:58:15.233416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.233406    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd" event={"ID":"12f7a25a-f059-4817-a209-e91edf3115b0","Type":"ContainerDied","Data":"cdd06dea7aa87fb6f7ba5e621ad94a8739e6b03b9399b36288a69336c592ad45"}
Nov 02 07:58:15.233524 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.233433    4515 scope.go:122] "RemoveContainer" containerID="fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"
Nov 02 07:58:15.248576 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.248557    4515 scope.go:122] "RemoveContainer" containerID="fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"
Nov 02 07:58:15.249739 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:58:15.249714    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2\": container with ID starting with fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2 not found: ID does not exist" containerID="fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"
Nov 02 07:58:15.249877 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.249769    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2"} err="failed to get container status \"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2\": rpc error: code = NotFound desc = could not find container \"fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2\": container with ID starting with fe0286495e1412225c20d7ce50e5c18c9953f873ea83c6c92e2def1bdccc87c2 not found: ID does not exist"
Nov 02 07:58:15.289947 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.289915    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:58:15.296067 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.296035    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-ct9fd"]
Nov 02 07:58:15.304193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.304164    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq4q\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-kube-api-access-gwq4q\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.304193 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.304191    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f7a25a-f059-4817-a209-e91edf3115b0-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.304397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.304201    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.304397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.304210    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f7a25a-f059-4817-a209-e91edf3115b0-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.304397 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.304219    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f7a25a-f059-4817-a209-e91edf3115b0-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:15.374935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:15.374900    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" path="/var/lib/kubelet/pods/12f7a25a-f059-4817-a209-e91edf3115b0/volumes"
Nov 02 07:58:17.524977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:17.524948    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:58:17.525501 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:17.525157    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy" containerID="cri-o://c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2" gracePeriod=30
Nov 02 07:58:17.724672 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:58:17.724583    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:58:22.817617 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.817590    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:58:22.927042 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927006    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927086    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927149    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927234 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927215    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927253    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927430 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927385    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927490    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:22.927535 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927517    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927641 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927560    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927644    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\") pod \"0c77bbef-ebae-4395-bad7-91ba03f47334\" (UID: \"0c77bbef-ebae-4395-bad7-91ba03f47334\") "
Nov 02 07:58:22.927752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927724    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:22.927884 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.927851    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:22.928073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928013    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:22.928073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928051    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:22.928454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928423    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c77bbef-ebae-4395-bad7-91ba03f47334-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:22.928454 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928454    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:22.928631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928468    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:22.928631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928481    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:22.928631 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.928496    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:22.929944 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.929922    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:22.930064 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.929936    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:22.930222 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.930205    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "kube-api-access-d6v94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:22.930285 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:22.930255    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo" pod "0c77bbef-ebae-4395-bad7-91ba03f47334" (UID: "0c77bbef-ebae-4395-bad7-91ba03f47334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:23.029969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.029933    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6v94\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-kube-api-access-d6v94\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:23.029969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.029968    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:23.029969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.029978    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:23.030218 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.029988    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c77bbef-ebae-4395-bad7-91ba03f47334-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:23.306729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.306693    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:58:23.356942 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.356904    4515 generic.go:356] "Generic (PLEG): container finished" podID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerID="c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2" exitCode=0
Nov 02 07:58:23.357147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.356951    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"
Nov 02 07:58:23.357147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.356985    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" event={"ID":"0c77bbef-ebae-4395-bad7-91ba03f47334","Type":"ContainerDied","Data":"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"}
Nov 02 07:58:23.357147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.357045    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g" event={"ID":"0c77bbef-ebae-4395-bad7-91ba03f47334","Type":"ContainerDied","Data":"6e7d230f17ad84e174b2a269f7a6406ef3d006724efbd2f045b5f23d1bb29910"}
Nov 02 07:58:23.357147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.357078    4515 scope.go:122] "RemoveContainer" containerID="c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"
Nov 02 07:58:23.368078 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.368054    4515 scope.go:122] "RemoveContainer" containerID="c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"
Nov 02 07:58:23.368371 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:58:23.368349    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2\": container with ID starting with c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2 not found: ID does not exist" containerID="c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"
Nov 02 07:58:23.368461 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.368378    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2"} err="failed to get container status \"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2\": rpc error: code = NotFound desc = could not find container \"c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2\": container with ID starting with c901c9b8a57fcdda111d539ebacaa25ca831c475d218debaccd617e0436c59e2 not found: ID does not exist"
Nov 02 07:58:23.412715 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.412680    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:58:23.416550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:23.416525    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-gbc5g"]
Nov 02 07:58:24.894073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:24.894015    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:58:24.894546 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:24.894277    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy" containerID="cri-o://60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f" gracePeriod=30
Nov 02 07:58:25.385671 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:25.385628    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" path="/var/lib/kubelet/pods/0c77bbef-ebae-4395-bad7-91ba03f47334/volumes"
Nov 02 07:58:30.169557 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.169493    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:58:30.268536 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.268502    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.268745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.268547    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.268745 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.268580    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.268883 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.268732    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.269050 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269006    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.269251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269095    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269257    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269285    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269354    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269397    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\") pod \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\" (UID: \"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d\") "
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269726    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.269924    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.270035 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270022    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.270498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270169    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:30.270780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270636    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.270780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270666    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.270780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270684    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.270780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270698    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.270780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.270711    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.271870 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.271828    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:30.272126 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.272085    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.272496 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.272463    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "kube-api-access-s58zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.272831 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.272802    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token" pod "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" (UID: "e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.372199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.372162    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.372199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.372194    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.372199 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.372204    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.372498 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.372214    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58zh\" (UniqueName: \"kubernetes.io/projected/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d-kube-api-access-s58zh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.481023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.480938    4515 generic.go:356] "Generic (PLEG): container finished" podID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerID="60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f" exitCode=0
Nov 02 07:58:30.481023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.480993    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"
Nov 02 07:58:30.481216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.481017    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" event={"ID":"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d","Type":"ContainerDied","Data":"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"}
Nov 02 07:58:30.481216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.481078    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t" event={"ID":"e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d","Type":"ContainerDied","Data":"7fb9a28ccef2c828b072d4cdb27b7069fb2ca4ceae585ea3ed92fa4c41d5cb91"}
Nov 02 07:58:30.481216 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.481115    4515 scope.go:122] "RemoveContainer" containerID="60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"
Nov 02 07:58:30.496774 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.496750    4515 scope.go:122] "RemoveContainer" containerID="60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"
Nov 02 07:58:30.497257 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:58:30.497211    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f\": container with ID starting with 60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f not found: ID does not exist" containerID="60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"
Nov 02 07:58:30.497409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.497252    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f"} err="failed to get container status \"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f\": rpc error: code = NotFound desc = could not find container \"60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f\": container with ID starting with 60cdbd02d756f1b306410d8bd6fa3c2f5169e6c3c7683a3ce9a89786159b205f not found: ID does not exist"
Nov 02 07:58:30.535690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.535662    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:58:30.547217 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:30.547188    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-sb94t"]
Nov 02 07:58:31.374980 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:31.374945    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" path="/var/lib/kubelet/pods/e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d/volumes"
Nov 02 07:58:47.724794 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:58:47.724757    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:58:49.539835 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:49.539794    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:58:54.278754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:58:54.278704    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:59:04.562583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.562542    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" podUID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" containerName="main" containerID="cri-o://472e46d51b6756570235608932e5c8cef991827a34386c46c2b8f2ad2b89a509" gracePeriod=45
Nov 02 07:59:04.727903 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.727869    4515 generic.go:356] "Generic (PLEG): container finished" podID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" containerID="472e46d51b6756570235608932e5c8cef991827a34386c46c2b8f2ad2b89a509" exitCode=0
Nov 02 07:59:04.728074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.728018    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" event={"ID":"f191ed8f-210e-46ea-be8d-3cdf2d6a0629","Type":"ContainerDied","Data":"472e46d51b6756570235608932e5c8cef991827a34386c46c2b8f2ad2b89a509"}
Nov 02 07:59:04.862609 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.862582    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:59:04.915543 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915512    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915576    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\" (UniqueName: \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915686    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\" (UniqueName: \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915731    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915921    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.915953    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\") pod \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\" (UID: \"f191ed8f-210e-46ea-be8d-3cdf2d6a0629\") "
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916205    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916426    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:04.916499 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916468    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:04.917015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916860    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:04.917015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916879    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:04.917015 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.916893    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:04.918979 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.918942    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:04.919112 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.919002    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:04.919574 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:04.919547    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q" pod "f191ed8f-210e-46ea-be8d-3cdf2d6a0629" (UID: "f191ed8f-210e-46ea-be8d-3cdf2d6a0629"). InnerVolumeSpecName "kube-api-access-tcq4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:05.018320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.018275    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:05.018320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.018319    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:05.018320 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.018331    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcq4q\" (UniqueName: \"kubernetes.io/projected/f191ed8f-210e-46ea-be8d-3cdf2d6a0629-kube-api-access-tcq4q\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:05.736752 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.736679    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"
Nov 02 07:59:05.737176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.736680    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8" event={"ID":"f191ed8f-210e-46ea-be8d-3cdf2d6a0629","Type":"ContainerDied","Data":"d7d1ca7998739bc6ae9bd502b042f2466944a7a0aba85a9b3c0f806356754af8"}
Nov 02 07:59:05.737176 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.736812    4515 scope.go:122] "RemoveContainer" containerID="472e46d51b6756570235608932e5c8cef991827a34386c46c2b8f2ad2b89a509"
Nov 02 07:59:05.790438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.790394    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:59:05.803282 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:05.803244    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/auth-test-model-9877ce-kserve-6d766b4f57-zhhh8"]
Nov 02 07:59:07.375042 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:07.375008    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" path="/var/lib/kubelet/pods/f191ed8f-210e-46ea-be8d-3cdf2d6a0629/volumes"
Nov 02 07:59:09.301091 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.301011    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerName="main" containerID="cri-o://ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365" gracePeriod=45
Nov 02 07:59:09.403709 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.403669    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.203:8000/ready\": dial tcp 10.133.0.203:8000: connect: connection refused"
Nov 02 07:59:09.573846 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.573815    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:59:09.690537 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690503    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.690758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690564    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\" (UniqueName: \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.690758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690626    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.690758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690681    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\" (UniqueName: \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.690927 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690812    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:09.690927 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690855    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.690927 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.690909    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\") pod \"21206eb1-2807-4f59-9936-65c47bb9ea56\" (UID: \"21206eb1-2807-4f59-9936-65c47bb9ea56\") "
Nov 02 07:59:09.691114 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.691076    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:09.691174 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.691131    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:09.691673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.691652    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.691734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.691681    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.691734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.691695    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.692976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.692950    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:09.693073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.692992    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "kube-api-access-9s4jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:09.693073 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.693030    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs" pod "21206eb1-2807-4f59-9936-65c47bb9ea56" (UID: "21206eb1-2807-4f59-9936-65c47bb9ea56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:09.766889 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.766852    4515 generic.go:356] "Generic (PLEG): container finished" podID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerID="ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365" exitCode=0
Nov 02 07:59:09.767069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.766900    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"
Nov 02 07:59:09.767069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.766934    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" event={"ID":"21206eb1-2807-4f59-9936-65c47bb9ea56","Type":"ContainerDied","Data":"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"}
Nov 02 07:59:09.767069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.766982    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt" event={"ID":"21206eb1-2807-4f59-9936-65c47bb9ea56","Type":"ContainerDied","Data":"aa31ee4434494aee4c7cb2239b6b86415f43dfc9baabddf2087853b0430ae42a"}
Nov 02 07:59:09.767069 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.767006    4515 scope.go:122] "RemoveContainer" containerID="ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"
Nov 02 07:59:09.777792 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.777771    4515 scope.go:122] "RemoveContainer" containerID="ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"
Nov 02 07:59:09.778106 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:09.778087    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365\": container with ID starting with ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365 not found: ID does not exist" containerID="ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"
Nov 02 07:59:09.778157 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.778117    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365"} err="failed to get container status \"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365\": rpc error: code = NotFound desc = could not find container \"ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365\": container with ID starting with ab2a64624d7d6b79623c58b453c967744129ad8a12cecd109ebb2c90a6392365 not found: ID does not exist"
Nov 02 07:59:09.793147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.793120    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21206eb1-2807-4f59-9936-65c47bb9ea56-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.793147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.793147    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21206eb1-2807-4f59-9936-65c47bb9ea56-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.793362 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.793160    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4jz\" (UniqueName: \"kubernetes.io/projected/21206eb1-2807-4f59-9936-65c47bb9ea56-kube-api-access-9s4jz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:09.824705 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.824625    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:59:09.829434 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:09.829403    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/auth-test-model-7f19e8-kserve-79945cfcc-4bmbt"]
Nov 02 07:59:10.177451 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:10.177368    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:59:11.435359 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:11.431946    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" path="/var/lib/kubelet/pods/21206eb1-2807-4f59-9936-65c47bb9ea56/volumes"
Nov 02 07:59:16.168830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.168797    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 07:59:16.169534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169517    4515 cpu_manager.go:414] "containerMap: removing container" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy"
Nov 02 07:59:16.169534 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169535    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169549    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" containerName="main"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169554    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" containerName="main"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169571    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169577    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169586    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169590    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169599    4515 cpu_manager.go:414] "containerMap: removing container" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerName="main"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169603    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerName="main"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169616    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c7528a6f-530c-43cc-88b3-f622254b43d5" containerName="payload-processing"
Nov 02 07:59:16.169637 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169620    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7528a6f-530c-43cc-88b3-f622254b43d5" containerName="payload-processing"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169698    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f7a25a-f059-4817-a209-e91edf3115b0" containerName="istio-proxy"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169708    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f191ed8f-210e-46ea-be8d-3cdf2d6a0629" containerName="main"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169715    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c77bbef-ebae-4395-bad7-91ba03f47334" containerName="istio-proxy"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169722    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d452c5-0a3d-4e6a-ae4a-f9166a6fe80d" containerName="istio-proxy"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169727    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21206eb1-2807-4f59-9936-65c47bb9ea56" containerName="main"
Nov 02 07:59:16.170003 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.169733    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7528a6f-530c-43cc-88b3-f622254b43d5" containerName="payload-processing"
Nov 02 07:59:16.173563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.173538    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.188982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.188949    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c712b1\""
Nov 02 07:59:16.201810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.201776    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c712b1-dockercfg-qcsg9\""
Nov 02 07:59:16.207364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.207331    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 07:59:16.346335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.322152    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.346335 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.322257    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvpl\" (UniqueName: \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.473508 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.473405    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvpl\" (UniqueName: \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.473700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.473600    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.477000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.474762    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.477365 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.477284    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 07:59:16.482550 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.482526    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 07:59:16.482655 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.482634    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.498134 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.498105    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvpl\" (UniqueName: \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\") pod \"payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.575376 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.575288    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rl5f\" (UniqueName: \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.575583 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.575388    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.676713 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.676671    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rl5f\" (UniqueName: \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.676935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.676721    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.677611 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.677588    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.702958 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.702921    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rl5f\" (UniqueName: \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\") pod \"payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:16.788874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.788843    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:16.796698 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:16.796670    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:17.168909 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:17.168877    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4d2d48_36c3_4144_9b19_a60aa603d91d.slice/crio-471672e0da690abc8cc2f5482035fe06aa3177d39d907e3731ecdd068c5740cd WatchSource:0}: Error finding container 471672e0da690abc8cc2f5482035fe06aa3177d39d907e3731ecdd068c5740cd: Status 404 returned error can't find the container with id 471672e0da690abc8cc2f5482035fe06aa3177d39d907e3731ecdd068c5740cd
Nov 02 07:59:17.169413 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.169389    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 07:59:17.186994 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:17.186950    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f35d01_275a_4261_a631_8d0e008f8760.slice/crio-384b8f6d8dda740b777e90c436f00ade5b4a230aa6eef8736561b51026f672b3 WatchSource:0}: Error finding container 384b8f6d8dda740b777e90c436f00ade5b4a230aa6eef8736561b51026f672b3: Status 404 returned error can't find the container with id 384b8f6d8dda740b777e90c436f00ade5b4a230aa6eef8736561b51026f672b3
Nov 02 07:59:17.188502 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.188479    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 07:59:17.728084 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:17.727998    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:59:17.881969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.881935    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" event={"ID":"ec4d2d48-36c3-4144-9b19-a60aa603d91d","Type":"ContainerStarted","Data":"7b6128bf20229a4c0bc21e349f5a13c3027dc23c70963e134d3903a031e1908c"}
Nov 02 07:59:17.882170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.882139    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:17.882231 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.882184    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" event={"ID":"ec4d2d48-36c3-4144-9b19-a60aa603d91d","Type":"ContainerStarted","Data":"471672e0da690abc8cc2f5482035fe06aa3177d39d907e3731ecdd068c5740cd"}
Nov 02 07:59:17.883572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.883545    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" event={"ID":"73f35d01-275a-4261-a631-8d0e008f8760","Type":"ContainerStarted","Data":"eede39c256180ce1aeea39dfc6cc5053efb5e125239297894561857c5dbad4b3"}
Nov 02 07:59:17.883700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.883592    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" event={"ID":"73f35d01-275a-4261-a631-8d0e008f8760","Type":"ContainerStarted","Data":"384b8f6d8dda740b777e90c436f00ade5b4a230aa6eef8736561b51026f672b3"}
Nov 02 07:59:17.883700 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.883615    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:17.926136 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:17.926085    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" podStartSLOduration=1.926072922 podStartE2EDuration="1.926072922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:17.92273677 +0000 UTC m=+2102.685173677" watchObservedRunningTime="2026-11-02 07:59:17.926072922 +0000 UTC m=+2102.688509827"
Nov 02 07:59:18.880718 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:18.880661    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" podStartSLOduration=2.880646337 podStartE2EDuration="2.880646337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:17.967883844 +0000 UTC m=+2102.730320750" watchObservedRunningTime="2026-11-02 07:59:18.880646337 +0000 UTC m=+2103.643083281"
Nov 02 07:59:18.881475 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:18.881455    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:59:18.881696 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:18.881677    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy" containerID="cri-o://0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054" gracePeriod=30
Nov 02 07:59:19.007085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:19.007048    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:59:20.094278 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:20.094244    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:59:24.244606 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.244577    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:59:24.365106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365064    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365119    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365138    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365188    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365299 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365271    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365380    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:24.365586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365390    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365536    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365586 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365575    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.365832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365627    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:24.365832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365641    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:24.365832 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365674    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\") pod \"1592d12f-65cb-41d7-8394-42832c403973\" (UID: \"1592d12f-65cb-41d7-8394-42832c403973\") "
Nov 02 07:59:24.366019 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.365918    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:24.366088 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366009    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:24.366459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366435    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.366577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366462    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.366577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366476    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1592d12f-65cb-41d7-8394-42832c403973-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.366577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366490    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.366577 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.366503    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.367604 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.367576    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "kube-api-access-2vxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:24.367833 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.367799    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:24.367955 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.367864    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:24.368052 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.367950    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token" pod "1592d12f-65cb-41d7-8394-42832c403973" (UID: "1592d12f-65cb-41d7-8394-42832c403973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:24.468325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.468218    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.468325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.468245    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1592d12f-65cb-41d7-8394-42832c403973-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.468325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.468256    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1592d12f-65cb-41d7-8394-42832c403973-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.468325 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.468265    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxbm\" (UniqueName: \"kubernetes.io/projected/1592d12f-65cb-41d7-8394-42832c403973-kube-api-access-2vxbm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:24.934736 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.934702    4515 generic.go:356] "Generic (PLEG): container finished" podID="1592d12f-65cb-41d7-8394-42832c403973" containerID="0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054" exitCode=0
Nov 02 07:59:24.934916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.934750    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"
Nov 02 07:59:24.934916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.934788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" event={"ID":"1592d12f-65cb-41d7-8394-42832c403973","Type":"ContainerDied","Data":"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"}
Nov 02 07:59:24.934916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.934848    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9" event={"ID":"1592d12f-65cb-41d7-8394-42832c403973","Type":"ContainerDied","Data":"cbf02903efd3d4563450b46877a9c681c5c9c5f7acfd6a3518d434ca58f65435"}
Nov 02 07:59:24.934916 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.934875    4515 scope.go:122] "RemoveContainer" containerID="0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"
Nov 02 07:59:24.946592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.946572    4515 scope.go:122] "RemoveContainer" containerID="0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"
Nov 02 07:59:24.946911 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:24.946887    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054\": container with ID starting with 0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054 not found: ID does not exist" containerID="0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"
Nov 02 07:59:24.947010 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:24.946916    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054"} err="failed to get container status \"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054\": rpc error: code = NotFound desc = could not find container \"0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054\": container with ID starting with 0cbf3500029e049323c766c08a2f6c123ab8c98b083e65aed888df2e7ee30054 not found: ID does not exist"
Nov 02 07:59:25.007572 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:25.007536    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:59:25.012358 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:25.012329    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-8vbb9"]
Nov 02 07:59:25.375147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:25.375109    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1592d12f-65cb-41d7-8394-42832c403973" path="/var/lib/kubelet/pods/1592d12f-65cb-41d7-8394-42832c403973/volumes"
Nov 02 07:59:28.613570 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.613322    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 07:59:28.620364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.614698    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy"
Nov 02 07:59:28.620364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.614739    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy"
Nov 02 07:59:28.620364 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.614922    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1592d12f-65cb-41d7-8394-42832c403973" containerName="istio-proxy"
Nov 02 07:59:28.621681 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.621659    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 07:59:28.621976 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.621958    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.657037 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.657007    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-696c7a\""
Nov 02 07:59:28.657263 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.657008    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-696c7a-dockercfg-v4q25\""
Nov 02 07:59:28.681928 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.681892    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.682117 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.682093    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rvj\" (UniqueName: \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.786352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.786296    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.786758 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.786737    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rvj\" (UniqueName: \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.787827 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.787632    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.810699 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.810669    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rvj\" (UniqueName: \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\") pod \"payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:28.895999 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.895906    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 07:59:28.896180 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.896151    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 07:59:28.941267 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:28.941233    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:29.270183 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:29.270147    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74f42a_49c1_4d00_be11_40e7d9656da4.slice/crio-21a67f6fe34dd9f434d9dc6fa2fdb26c1c39d922ca2d574a9aa26112b4bbe9c1 WatchSource:0}: Error finding container 21a67f6fe34dd9f434d9dc6fa2fdb26c1c39d922ca2d574a9aa26112b4bbe9c1: Status 404 returned error can't find the container with id 21a67f6fe34dd9f434d9dc6fa2fdb26c1c39d922ca2d574a9aa26112b4bbe9c1
Nov 02 07:59:29.271697 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:29.271671    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 07:59:29.988119 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:29.988074    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" event={"ID":"ed74f42a-49c1-4d00-be11-40e7d9656da4","Type":"ContainerStarted","Data":"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"}
Nov 02 07:59:29.988620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:29.988142    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" event={"ID":"ed74f42a-49c1-4d00-be11-40e7d9656da4","Type":"ContainerStarted","Data":"21a67f6fe34dd9f434d9dc6fa2fdb26c1c39d922ca2d574a9aa26112b4bbe9c1"}
Nov 02 07:59:29.988620 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:29.988284    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:34.029673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.029630    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" podUID="c466660d-76e7-4878-9ba5-5063c956691b" containerName="main" containerID="cri-o://cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7" gracePeriod=45
Nov 02 07:59:34.316721 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.316693    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:59:34.361120 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.361042    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" podStartSLOduration=6.361027148 podStartE2EDuration="6.361027148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:30.03492206 +0000 UTC m=+2114.797358965" watchObservedRunningTime="2026-11-02 07:59:34.361027148 +0000 UTC m=+2119.123464053"
Nov 02 07:59:34.413731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.413687    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.413963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.413812    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.413963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.413845    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.413963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.413938    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\" (UniqueName: \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.414113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414074    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.414113 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414070    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:34.414194 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414116    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:34.414229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414208    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\") pod \"c466660d-76e7-4878-9ba5-5063c956691b\" (UID: \"c466660d-76e7-4878-9ba5-5063c956691b\") "
Nov 02 07:59:34.414416 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414385    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:34.414728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414707    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:34.414851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414731    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:34.414851 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.414740    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:34.416437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.416412    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:34.416437 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.416425    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:34.416590 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.416444    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m" pod "c466660d-76e7-4878-9ba5-5063c956691b" (UID: "c466660d-76e7-4878-9ba5-5063c956691b"). InnerVolumeSpecName "kube-api-access-tbh8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:34.516106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.516075    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c466660d-76e7-4878-9ba5-5063c956691b-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:34.516106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.516103    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbh8m\" (UniqueName: \"kubernetes.io/projected/c466660d-76e7-4878-9ba5-5063c956691b-kube-api-access-tbh8m\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:34.516106 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:34.516113    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c466660d-76e7-4878-9ba5-5063c956691b-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.068662 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.068632    4515 generic.go:356] "Generic (PLEG): container finished" podID="c466660d-76e7-4878-9ba5-5063c956691b" containerID="cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7" exitCode=0
Nov 02 07:59:35.069129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.068672    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"
Nov 02 07:59:35.069129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.068716    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" event={"ID":"c466660d-76e7-4878-9ba5-5063c956691b","Type":"ContainerDied","Data":"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"}
Nov 02 07:59:35.069129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.068760    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" event={"ID":"c466660d-76e7-4878-9ba5-5063c956691b","Type":"ContainerDied","Data":"9c6a670c7f62f6b2a013b18a884a700e2bb972022f64b498212fa65bce1b10b2"}
Nov 02 07:59:35.069129 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.068778    4515 scope.go:122] "RemoveContainer" containerID="cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"
Nov 02 07:59:35.082436 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.082418    4515 scope.go:122] "RemoveContainer" containerID="cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"
Nov 02 07:59:35.082708 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:35.082687    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7\": container with ID starting with cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7 not found: ID does not exist" containerID="cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"
Nov 02 07:59:35.082780 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.082720    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7"} err="failed to get container status \"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7\": rpc error: code = NotFound desc = could not find container \"cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7\": container with ID starting with cec799d5d1ed385be13569ab525d7369b3363a147419ac323599d8544603a1b7 not found: ID does not exist"
Nov 02 07:59:35.118207 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.118163    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" podUID="854e735d-24d2-4025-9a01-2195b2dfab4e" containerName="main" containerID="cri-o://10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4" gracePeriod=45
Nov 02 07:59:35.153445 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.152390    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:59:35.175558 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.175485    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn"]
Nov 02 07:59:35.303822 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.303785    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-12df3c/auth-test-model-12df3c-kserve-b567bcdcc-w46jn" podUID="c466660d-76e7-4878-9ba5-5063c956691b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:59:35.374967 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.374895    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c466660d-76e7-4878-9ba5-5063c956691b" path="/var/lib/kubelet/pods/c466660d-76e7-4878-9ba5-5063c956691b/volumes"
Nov 02 07:59:35.408817 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.408792    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:59:35.547529 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547497    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.547729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547628    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.547729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547689    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.547869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547808    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:35.547869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547841    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.547869 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547865    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\" (UniqueName: \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.548055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547919    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\") pod \"854e735d-24d2-4025-9a01-2195b2dfab4e\" (UID: \"854e735d-24d2-4025-9a01-2195b2dfab4e\") "
Nov 02 07:59:35.548055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547935    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:35.548055 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.547955    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:35.548447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.548422    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.548447 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.548447    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.548651 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.548456    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.550169 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.550143    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:35.550290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.550187    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "kube-api-access-r5w7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:35.550290 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.550214    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs" pod "854e735d-24d2-4025-9a01-2195b2dfab4e" (UID: "854e735d-24d2-4025-9a01-2195b2dfab4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:35.650147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.650071    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e735d-24d2-4025-9a01-2195b2dfab4e-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.650147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.650101    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w7b\" (UniqueName: \"kubernetes.io/projected/854e735d-24d2-4025-9a01-2195b2dfab4e-kube-api-access-r5w7b\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:35.650147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:35.650111    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e735d-24d2-4025-9a01-2195b2dfab4e-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:36.078497 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.078463    4515 generic.go:356] "Generic (PLEG): container finished" podID="854e735d-24d2-4025-9a01-2195b2dfab4e" containerID="10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4" exitCode=0
Nov 02 07:59:36.078997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.078532    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"
Nov 02 07:59:36.078997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.078593    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" event={"ID":"854e735d-24d2-4025-9a01-2195b2dfab4e","Type":"ContainerDied","Data":"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"}
Nov 02 07:59:36.078997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.078638    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k" event={"ID":"854e735d-24d2-4025-9a01-2195b2dfab4e","Type":"ContainerDied","Data":"bd35f622ed07ed7ee3a3de194f8012beab998fcaadd0798308868a78b09f8e3d"}
Nov 02 07:59:36.078997 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.078657    4515 scope.go:122] "RemoveContainer" containerID="10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"
Nov 02 07:59:36.090058 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.090033    4515 scope.go:122] "RemoveContainer" containerID="10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"
Nov 02 07:59:36.090374 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:36.090352    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4\": container with ID starting with 10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4 not found: ID does not exist" containerID="10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"
Nov 02 07:59:36.090487 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.090379    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4"} err="failed to get container status \"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4\": rpc error: code = NotFound desc = could not find container \"10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4\": container with ID starting with 10c405bbdd53cd4dabb4bc0e6d93a2e45dec942834e312881054a6938e1459c4 not found: ID does not exist"
Nov 02 07:59:36.141251 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.141215    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:59:36.153492 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:36.153461    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/auth-test-model-56bbb6-kserve-6d8d87c6df-5zp2k"]
Nov 02 07:59:37.375906 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:37.375866    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854e735d-24d2-4025-9a01-2195b2dfab4e" path="/var/lib/kubelet/pods/854e735d-24d2-4025-9a01-2195b2dfab4e/volumes"
Nov 02 07:59:38.294452 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.294062    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 07:59:38.307020 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.303270    4515 cpu_manager.go:414] "containerMap: removing container" podUID="854e735d-24d2-4025-9a01-2195b2dfab4e" containerName="main"
Nov 02 07:59:38.309417 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.309381    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854e735d-24d2-4025-9a01-2195b2dfab4e" containerName="main"
Nov 02 07:59:38.309595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.309480    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c466660d-76e7-4878-9ba5-5063c956691b" containerName="main"
Nov 02 07:59:38.309595 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.309493    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c466660d-76e7-4878-9ba5-5063c956691b" containerName="main"
Nov 02 07:59:38.309754 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.309737    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="854e735d-24d2-4025-9a01-2195b2dfab4e" containerName="main"
Nov 02 07:59:38.309826 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.309760    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c466660d-76e7-4878-9ba5-5063c956691b" containerName="main"
Nov 02 07:59:38.348340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.345590    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.355184 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.352572    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 07:59:38.371690 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.371660    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-7f38f1-openshift-default-dockercfg-q7vv9\""
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527332    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527386    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527602    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527673    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527706    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527766    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527807    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527832    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5m6j\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.529353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.527868    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629335    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5m6j\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629425 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629384    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629443    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629466    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629597    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629675 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629663    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629691    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.629911 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629899    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630026 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.629924    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.630151    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630175 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.630166    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630322 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.630234    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630322 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.630262    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.630810 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.630786    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.632776 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.632754    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.632982 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.632959    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.669367 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.669320    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5m6j\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.673471 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.673446    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\") pod \"e2e-noten-7f38f1-openshift-default-59c89b95-f78n4\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") " pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:38.967740 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:38.967651    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:39.306246 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:39.306211    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c196c76_fa6a_49d0_8c5f_d9f1959f75e0.slice/crio-16410efd5421dc654e08856206b2118bfefc36a4044e73741d001b4c35eb7bc9 WatchSource:0}: Error finding container 16410efd5421dc654e08856206b2118bfefc36a4044e73741d001b4c35eb7bc9: Status 404 returned error can't find the container with id 16410efd5421dc654e08856206b2118bfefc36a4044e73741d001b4c35eb7bc9
Nov 02 07:59:39.308049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:39.308022    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 07:59:39.308935 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:39.308898    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:39.309056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:39.308977    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:39.309056 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:39.309022    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:40.115546 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:40.115499    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" event={"ID":"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0","Type":"ContainerStarted","Data":"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"}
Nov 02 07:59:40.116013 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:40.115567    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" event={"ID":"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0","Type":"ContainerStarted","Data":"16410efd5421dc654e08856206b2118bfefc36a4044e73741d001b4c35eb7bc9"}
Nov 02 07:59:40.194782 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:40.194733    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" podStartSLOduration=2.194717623 podStartE2EDuration="2.194717623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:40.179605186 +0000 UTC m=+2124.942042091" watchObservedRunningTime="2026-11-02 07:59:40.194717623 +0000 UTC m=+2124.957154529"
Nov 02 07:59:40.968229 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:40.968192    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:40.974049 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:40.974022    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:41.026380 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:41.026341    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 07:59:41.123219 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:41.123190    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:41.124204 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:41.124180    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 07:59:47.723869 ip-10-0-137-156 kubenswrapper[4515]: E1102 07:59:47.723834    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 07:59:48.988824 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:48.988791    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:59:48.989297 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:48.989005    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy" containerID="cri-o://aa450f58cbd1da716c6c99bfddd15d0fb660b3c21ca24c249023c92f7b3f6919" gracePeriod=30
Nov 02 07:59:52.572071 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.572034    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 07:59:52.576717 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.576695    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.611794 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.611756    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 07:59:52.641072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641037    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641100    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641255 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641225    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641272    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxwf\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641289    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641333    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641465    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641592 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641493    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.641673 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.641636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.643223 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.643203    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3575b9-openshift-default-dockercfg-6pv8l\""
Nov 02 07:59:52.743200 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743162    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743221    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743264    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743302    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxwf\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743359    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743438 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743392    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743448    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743478    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.743731 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743592    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.744085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743764    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.744085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743849    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.744085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.743895    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.744085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.744001    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.744260 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.744206    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.745998 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.745976    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.746201 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.746181    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.768025 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.767985    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.768186 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.768050    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxwf\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:52.893427 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:52.893343    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:53.214207 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:53.214173    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151f5506_8b72_459f_ba40_4479c93ca1c5.slice/crio-b138f0f96a3f1632e81a23b3ec450a20bce01ce68635c55f933d79cad6a6956c WatchSource:0}: Error finding container b138f0f96a3f1632e81a23b3ec450a20bce01ce68635c55f933d79cad6a6956c: Status 404 returned error can't find the container with id b138f0f96a3f1632e81a23b3ec450a20bce01ce68635c55f933d79cad6a6956c
Nov 02 07:59:53.215668 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:53.215644    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 07:59:53.216969 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:53.216931    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:53.217040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:53.216997    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:53.217040 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:53.217026    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:53.230173 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:53.230146    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" event={"ID":"151f5506-8b72-459f-ba40-4479c93ca1c5","Type":"ContainerStarted","Data":"b138f0f96a3f1632e81a23b3ec450a20bce01ce68635c55f933d79cad6a6956c"}
Nov 02 07:59:54.239097 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.239059    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" event={"ID":"151f5506-8b72-459f-ba40-4479c93ca1c5","Type":"ContainerStarted","Data":"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"}
Nov 02 07:59:54.240652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.240494    4515 generic.go:356] "Generic (PLEG): container finished" podID="60a87237-365f-4061-ad26-9865b4accd23" containerID="aa450f58cbd1da716c6c99bfddd15d0fb660b3c21ca24c249023c92f7b3f6919" exitCode=0
Nov 02 07:59:54.240652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.240559    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" event={"ID":"60a87237-365f-4061-ad26-9865b4accd23","Type":"ContainerDied","Data":"aa450f58cbd1da716c6c99bfddd15d0fb660b3c21ca24c249023c92f7b3f6919"}
Nov 02 07:59:54.240652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.240597    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm" event={"ID":"60a87237-365f-4061-ad26-9865b4accd23","Type":"ContainerDied","Data":"25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a"}
Nov 02 07:59:54.240652 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.240613    4515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f1b442ed0ebf9be34c91be4cdb5eb6d3c6a8bd9796c4d3dcacbbc30c4a411a"
Nov 02 07:59:54.261957 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.261931    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:59:54.362023 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.361923    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podStartSLOduration=2.3619069169999998 podStartE2EDuration="2.361906917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:54.30062439 +0000 UTC m=+2139.063061314" watchObservedRunningTime="2026-11-02 07:59:54.361906917 +0000 UTC m=+2139.124343822"
Nov 02 07:59:54.364638 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364593    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.364798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364679    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.364798 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364714    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.364885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364805    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.364946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364901    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.364946 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364904    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:54.365066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364982    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.365066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.364989    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:54.365066 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365049    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.365243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365117    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.365243 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365226    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\") pod \"60a87237-365f-4061-ad26-9865b4accd23\" (UID: \"60a87237-365f-4061-ad26-9865b4accd23\") "
Nov 02 07:59:54.365724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365493    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:54.365724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365557    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:54.365724 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.365689    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:54.366203 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.366180    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a87237-365f-4061-ad26-9865b4accd23-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.366451 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.366432    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.366528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.366458    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.366528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.366473    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.366528 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.366487    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.367663 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.367638    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "kube-api-access-w7jm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:54.368459 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.368363    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:54.368847 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.368823    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:54.370538 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.370514    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo" pod "60a87237-365f-4061-ad26-9865b4accd23" (UID: "60a87237-365f-4061-ad26-9865b4accd23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:54.468000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.467874    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a87237-365f-4061-ad26-9865b4accd23-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.468000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.467913    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.468000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.467926    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jm8\" (UniqueName: \"kubernetes.io/projected/60a87237-365f-4061-ad26-9865b4accd23-kube-api-access-w7jm8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.468000 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.467956    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a87237-365f-4061-ad26-9865b4accd23-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 07:59:54.894353 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.894247    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 07:59:54.896085 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.896055    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:54.896237 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:54.896124    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 07:59:55.152728 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.152642    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 07:59:55.153352 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.153336    4515 cpu_manager.go:414] "containerMap: removing container" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy"
Nov 02 07:59:55.153409 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.153354    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy"
Nov 02 07:59:55.153553 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.153530    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a87237-365f-4061-ad26-9865b4accd23" containerName="istio-proxy"
Nov 02 07:59:55.157597 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.157574    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.187610 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.187574    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-6929a4-openshift-default-dockercfg-dv4x2\""
Nov 02 07:59:55.193065 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.193027    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 07:59:55.252977 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.252948    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"
Nov 02 07:59:55.279304 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279270    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279332    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279395    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279470 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279421    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279578 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279496    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279615 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279581    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279683 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279663    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4sn\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279729 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279716    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.279785 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.279749    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.345489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.345438    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:59:55.349074 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.349046    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-4rlvm"]
Nov 02 07:59:55.375830 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.375790    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a87237-365f-4061-ad26-9865b4accd23" path="/var/lib/kubelet/pods/60a87237-365f-4061-ad26-9865b4accd23/volumes"
Nov 02 07:59:55.381682 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.381651    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.381874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.381764    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.381874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.381827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4sn\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.381874 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.381853    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.381900    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382100    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382128    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382147 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382141    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382191    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.382406 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382227    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.383170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382697    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.383170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.382846    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.383170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.383126    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.383170 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.383157    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.387640 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.387613    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.387885 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.387865    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.408424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.408356    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4sn\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.408424 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.408374    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.472076 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.472038    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:55.843734 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.843684    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 07:59:55.845375 ip-10-0-137-156 kubenswrapper[4515]: W1102 07:59:55.845339    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba27247_c0eb_427b_9a7b_b765a793d789.slice/crio-ee975c54551a55bab85d3e2ff1ef3aea58cf777b30f11b4dcdfa6f15bd74987f WatchSource:0}: Error finding container ee975c54551a55bab85d3e2ff1ef3aea58cf777b30f11b4dcdfa6f15bd74987f: Status 404 returned error can't find the container with id ee975c54551a55bab85d3e2ff1ef3aea58cf777b30f11b4dcdfa6f15bd74987f
Nov 02 07:59:55.848354 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.848278    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:55.848432 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.848400    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:55.848468 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.848438    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:55.894513 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.894483    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:55.894678 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:55.894541    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 07:59:56.267072 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.267035    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" event={"ID":"aba27247-c0eb-427b-9a7b-b765a793d789","Type":"ContainerStarted","Data":"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"}
Nov 02 07:59:56.267495 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.267092    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" event={"ID":"aba27247-c0eb-427b-9a7b-b765a793d789","Type":"ContainerStarted","Data":"ee975c54551a55bab85d3e2ff1ef3aea58cf777b30f11b4dcdfa6f15bd74987f"}
Nov 02 07:59:56.473403 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.473231    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 07:59:56.493777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.474725    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 07:59:56.493777 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.474787    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 07:59:56.894192 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.894154    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:56.894622 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:56.894578    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 07:59:57.472648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:57.472613    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 07:59:57.473148 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:57.472660    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 07:59:57.894489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:57.894112    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:57.894489 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:57.894161    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 07:59:58.484336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:58.476779    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 07:59:58.484336 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:58.476844    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 07:59:58.893735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:58.893647    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:58.893735 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:58.893694    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 07:59:59.016648 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.016574    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podStartSLOduration=4.01654916 podStartE2EDuration="4.01654916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:56.346662967 +0000 UTC m=+2141.109099877" watchObservedRunningTime="2026-11-02 07:59:59.01654916 +0000 UTC m=+2143.778986066"
Nov 02 07:59:59.017340 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.017295    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 07:59:59.473374 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.473298    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 07:59:59.473601 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.473407    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 07:59:59.894563 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.894528    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:59:59.894963 ip-10-0-137-156 kubenswrapper[4515]: I1102 07:59:59.894582    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 08:00:00.343411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.336090    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:00.470297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.470264    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:00.482228 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.470876    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.482228 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.473194    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 08:00:00.482228 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.480546    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 08:00:00.488683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.488619    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:00.489120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.489101    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.501447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.501399    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:00.505036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.505004    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:00.532332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.509380    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:00:00.601380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.599342    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:00.601380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.599471    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.639715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.634929    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.639715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.635130    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.639715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.635357    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpfb\" (UniqueName: \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.639715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.635389    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nsv\" (UniqueName: \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.740331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740210    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.740543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740376    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.740543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740437    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclr5\" (UniqueName: \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.740543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.740726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740575    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpfb\" (UniqueName: \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.740726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.740601    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nsv\" (UniqueName: \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.748332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.742597    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.748332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.743272    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.777824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.777784    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpfb\" (UniqueName: \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.778292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.778271    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nsv\" (UniqueName: \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:00.853396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.851354    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclr5\" (UniqueName: \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.853396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.851444    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.853396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.852670    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.877503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.877466    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclr5\" (UniqueName: \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:00.906150 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.906115    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:00.933820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:00.930553    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:01.011414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.008564    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:01.087467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.085437    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 08:00:01.087467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.086598    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 08:00:01.097331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.093016    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 08:00:01.310157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.310115    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:01.310704 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:01.310677    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3ba74f_eb5d_411d_b8a3_be421ebf6206.slice/crio-7de9f378a10a9c38406e6b72334ab61e7630448f55081e8c5d8b21d9a8c3e15a WatchSource:0}: Error finding container 7de9f378a10a9c38406e6b72334ab61e7630448f55081e8c5d8b21d9a8c3e15a: Status 404 returned error can't find the container with id 7de9f378a10a9c38406e6b72334ab61e7630448f55081e8c5d8b21d9a8c3e15a
Nov 02 08:00:01.457865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.457056    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerStarted","Data":"7de9f378a10a9c38406e6b72334ab61e7630448f55081e8c5d8b21d9a8c3e15a"}
Nov 02 08:00:01.473855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.473518    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 08:00:01.473855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.473565    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 08:00:01.510784 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:01.510752    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c041a2_7d09_4166_b11e_b0036340cb8d.slice/crio-d37eda962d49607fdfbb11ca7b6c523dd2f0b790e73d36b94af2029cfb1936a2 WatchSource:0}: Error finding container d37eda962d49607fdfbb11ca7b6c523dd2f0b790e73d36b94af2029cfb1936a2: Status 404 returned error can't find the container with id d37eda962d49607fdfbb11ca7b6c523dd2f0b790e73d36b94af2029cfb1936a2
Nov 02 08:00:01.513303 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.513280    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:01.648751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:01.648714    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:01.662976 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:01.662942    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78b0c0f_59d2_4ddf_abb7_8652d59d3102.slice/crio-c99d975a084eac23838281d4411f2fd5f96ad1814e0f65228c242e9a75acba01 WatchSource:0}: Error finding container c99d975a084eac23838281d4411f2fd5f96ad1814e0f65228c242e9a75acba01: Status 404 returned error can't find the container with id c99d975a084eac23838281d4411f2fd5f96ad1814e0f65228c242e9a75acba01
Nov 02 08:00:02.461977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.461882    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerStarted","Data":"e8dec517762caa414ff538aa25de3751746bd9dd41f498739a4332237b7a64d4"}
Nov 02 08:00:02.461977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.461940    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerStarted","Data":"c99d975a084eac23838281d4411f2fd5f96ad1814e0f65228c242e9a75acba01"}
Nov 02 08:00:02.463679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.463644    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerStarted","Data":"aa1ab0a03c7a3afc5bc959a16ac520da888e103337d596b9c1bf29375a8a8941"}
Nov 02 08:00:02.463828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.463703    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerStarted","Data":"d37eda962d49607fdfbb11ca7b6c523dd2f0b790e73d36b94af2029cfb1936a2"}
Nov 02 08:00:02.465241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.465212    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerStarted","Data":"fdf1b4c77b2c62494da9e16ccb087f33fa934c5f65745585e3bac277b20cd46e"}
Nov 02 08:00:02.472920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.472894    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 08:00:02.473046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.472947    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 08:00:02.546447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.546159    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" podStartSLOduration=2.546138146 podStartE2EDuration="2.546138146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.542918012 +0000 UTC m=+2147.305354918" watchObservedRunningTime="2026-11-02 08:00:02.546138146 +0000 UTC m=+2147.308575053"
Nov 02 08:00:02.601292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.601239    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" podStartSLOduration=2.601225647 podStartE2EDuration="2.601225647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.597915764 +0000 UTC m=+2147.360352670" watchObservedRunningTime="2026-11-02 08:00:02.601225647 +0000 UTC m=+2147.363662609"
Nov 02 08:00:02.660393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.660337    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" podStartSLOduration=2.6603014209999998 podStartE2EDuration="2.660301421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.660052828 +0000 UTC m=+2147.422489733" watchObservedRunningTime="2026-11-02 08:00:02.660301421 +0000 UTC m=+2147.422738327"
Nov 02 08:00:02.836206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.836165    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 08:00:02.836484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:02.836455    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy" containerID="cri-o://35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c" gracePeriod=30
Nov 02 08:00:03.474268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.474233    4515 generic.go:356] "Generic (PLEG): container finished" podID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerID="e8dec517762caa414ff538aa25de3751746bd9dd41f498739a4332237b7a64d4" exitCode=60
Nov 02 08:00:03.474776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.474327    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerDied","Data":"e8dec517762caa414ff538aa25de3751746bd9dd41f498739a4332237b7a64d4"}
Nov 02 08:00:03.474776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.474656    4515 scope.go:122] "RemoveContainer" containerID="e8dec517762caa414ff538aa25de3751746bd9dd41f498739a4332237b7a64d4"
Nov 02 08:00:03.476351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.476286    4515 generic.go:356] "Generic (PLEG): container finished" podID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerID="aa1ab0a03c7a3afc5bc959a16ac520da888e103337d596b9c1bf29375a8a8941" exitCode=60
Nov 02 08:00:03.476486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.476369    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerDied","Data":"aa1ab0a03c7a3afc5bc959a16ac520da888e103337d596b9c1bf29375a8a8941"}
Nov 02 08:00:03.476895 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.476690    4515 scope.go:122] "RemoveContainer" containerID="aa1ab0a03c7a3afc5bc959a16ac520da888e103337d596b9c1bf29375a8a8941"
Nov 02 08:00:03.478364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.478297    4515 generic.go:356] "Generic (PLEG): container finished" podID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerID="fdf1b4c77b2c62494da9e16ccb087f33fa934c5f65745585e3bac277b20cd46e" exitCode=60
Nov 02 08:00:03.478364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.478339    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerDied","Data":"fdf1b4c77b2c62494da9e16ccb087f33fa934c5f65745585e3bac277b20cd46e"}
Nov 02 08:00:03.478750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.478668    4515 scope.go:122] "RemoveContainer" containerID="fdf1b4c77b2c62494da9e16ccb087f33fa934c5f65745585e3bac277b20cd46e"
Nov 02 08:00:03.479238 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.479220    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 08:00:03.480240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.480072    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 08:00:03.481183 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:03.481164    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 08:00:04.488267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.488231    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerStarted","Data":"e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"}
Nov 02 08:00:04.490362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.490336    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerStarted","Data":"fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"}
Nov 02 08:00:04.492423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.492395    4515 generic.go:356] "Generic (PLEG): container finished" podID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerID="028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f" exitCode=60
Nov 02 08:00:04.492549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.492474    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerDied","Data":"028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f"}
Nov 02 08:00:04.492549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.492526    4515 scope.go:122] "RemoveContainer" containerID="fdf1b4c77b2c62494da9e16ccb087f33fa934c5f65745585e3bac277b20cd46e"
Nov 02 08:00:04.492707 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:04.492689    4515 scope.go:122] "RemoveContainer" containerID="028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f"
Nov 02 08:00:04.492955 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:04.492936    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd_odh-ai-gateway-infra(bb3ba74f-eb5d-411d-b8a3-be421ebf6206)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206"
Nov 02 08:00:05.502531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.502500    4515 scope.go:122] "RemoveContainer" containerID="028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f"
Nov 02 08:00:05.502949 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:05.502873    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd_odh-ai-gateway-infra(bb3ba74f-eb5d-411d-b8a3-be421ebf6206)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206"
Nov 02 08:00:05.504155 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.504128    4515 generic.go:356] "Generic (PLEG): container finished" podID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerID="e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba" exitCode=60
Nov 02 08:00:05.504284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.504187    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerDied","Data":"e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"}
Nov 02 08:00:05.504284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.504246    4515 scope.go:122] "RemoveContainer" containerID="e8dec517762caa414ff538aa25de3751746bd9dd41f498739a4332237b7a64d4"
Nov 02 08:00:05.504494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.504476    4515 scope.go:122] "RemoveContainer" containerID="e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"
Nov 02 08:00:05.504812 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:05.504785    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf_odh-ai-gateway-infra(d78b0c0f-59d2-4ddf-abb7-8652d59d3102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102"
Nov 02 08:00:05.506260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.506240    4515 generic.go:356] "Generic (PLEG): container finished" podID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerID="fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace" exitCode=60
Nov 02 08:00:05.506393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.506339    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerDied","Data":"fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"}
Nov 02 08:00:05.506547 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.506531    4515 scope.go:122] "RemoveContainer" containerID="fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"
Nov 02 08:00:05.506839 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:05.506819    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz_odh-ai-gateway-infra(b7c041a2-7d09-4166-b11e-b0036340cb8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d"
Nov 02 08:00:05.524297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:05.524081    4515 scope.go:122] "RemoveContainer" containerID="aa1ab0a03c7a3afc5bc959a16ac520da888e103337d596b9c1bf29375a8a8941"
Nov 02 08:00:06.515124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:06.515083    4515 scope.go:122] "RemoveContainer" containerID="e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"
Nov 02 08:00:06.515577 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:06.515474    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf_odh-ai-gateway-infra(d78b0c0f-59d2-4ddf-abb7-8652d59d3102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102"
Nov 02 08:00:06.516679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:06.516655    4515 scope.go:122] "RemoveContainer" containerID="fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"
Nov 02 08:00:06.516996 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:06.516972    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz_odh-ai-gateway-infra(b7c041a2-7d09-4166-b11e-b0036340cb8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d"
Nov 02 08:00:06.942081 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:06.941983    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 08:00:06.942273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:06.942207    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" containerID="cri-o://26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196" gracePeriod=30
Nov 02 08:00:07.052091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:07.052056    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 08:00:07.052578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:07.052541    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerName="istio-proxy" containerID="cri-o://296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572" gracePeriod=30
Nov 02 08:00:07.959340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:07.959295    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 08:00:08.232150 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.232121    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 08:00:08.261532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261497    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261552    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261595    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261638    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261723    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261809    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261878    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261921    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.261997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.261947    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\") pod \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\" (UID: \"f4bba617-9ead-4c0a-9bb6-689b8c16de59\") "
Nov 02 08:00:08.262385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262294    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:08.262519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262398    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:08.262519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262499    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:08.262635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262587    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:08.262734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262706    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:08.262860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262793    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.262860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262815    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.262860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262829    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.262860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.262842    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.264340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.264293    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:00:08.264621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.264603    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "kube-api-access-mbddd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:08.264742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.264721    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:08.264782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.264749    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token" pod "f4bba617-9ead-4c0a-9bb6-689b8c16de59" (UID: "f4bba617-9ead-4c0a-9bb6-689b8c16de59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:08.364619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.364578    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.364619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.364609    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbddd\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-kube-api-access-mbddd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.364619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.364619    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.364619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.364630    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.364925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.364640    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bba617-9ead-4c0a-9bb6-689b8c16de59-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:08.596780 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.596734    4515 generic.go:356] "Generic (PLEG): container finished" podID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerID="35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c" exitCode=0
Nov 02 08:00:08.596969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.596802    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" event={"ID":"f4bba617-9ead-4c0a-9bb6-689b8c16de59","Type":"ContainerDied","Data":"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"}
Nov 02 08:00:08.596969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.596898    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc" event={"ID":"f4bba617-9ead-4c0a-9bb6-689b8c16de59","Type":"ContainerDied","Data":"4388f655e01fa9121a603080b925e8c21608fcb13f63b1fd601c1c047e8b410d"}
Nov 02 08:00:08.596969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.596915    4515 scope.go:122] "RemoveContainer" containerID="35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"
Nov 02 08:00:08.597181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.596813    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"
Nov 02 08:00:08.618607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.618576    4515 scope.go:122] "RemoveContainer" containerID="35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"
Nov 02 08:00:08.619042 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:08.619000    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c\": container with ID starting with 35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c not found: ID does not exist" containerID="35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"
Nov 02 08:00:08.619187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.619054    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c"} err="failed to get container status \"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c\": rpc error: code = NotFound desc = could not find container \"35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c\": container with ID starting with 35059cc3ca68eb667902a225ad99d748a3a8705c2cc0d42a290e4ec3eae0aa6c not found: ID does not exist"
Nov 02 08:00:08.657966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.657931    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 08:00:08.670478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:08.670446    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-vrrvc"]
Nov 02 08:00:09.376356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:09.376301    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" path="/var/lib/kubelet/pods/f4bba617-9ead-4c0a-9bb6-689b8c16de59/volumes"
Nov 02 08:00:11.125331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:11.125280    4515 patch_prober.go:29] interesting pod/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:00:11.125802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:11.125375    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:00:11.534280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:11.534244    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8fmqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:00:11.534486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:11.534318    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:00:12.245936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.245907    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 08:00:12.318638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318603    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318653    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318690    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318718    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318794    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318832    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.318928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.318863    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.319191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.319010    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.319191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.319148    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\") pod \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\" (UID: \"1e197b26-0b5d-40c3-b4ff-7f120cec07d9\") "
Nov 02 08:00:12.320049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.319965    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.320185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.320164    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.320273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.320245    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.320482 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.320255    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.320482 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.320409    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.320810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.320779    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:12.322141 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.322110    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.322495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.322462    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:00:12.323907 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.323883    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:12.325442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.325286    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm" pod "1e197b26-0b5d-40c3-b4ff-7f120cec07d9" (UID: "1e197b26-0b5d-40c3-b4ff-7f120cec07d9"). InnerVolumeSpecName "kube-api-access-l9fwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:12.353812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.353784    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 08:00:12.421181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421145    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421205    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421239    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421320    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421356    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421385    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421515    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421578    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.421740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421571    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.421740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421606    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421660    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\") pod \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\" (UID: \"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0\") "
Nov 02 08:00:12.421990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.421913    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.422483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422444    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422513    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422532    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422547    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-kube-api-access-l9fwm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422561    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422576    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422591    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422603    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422616    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.422623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422627    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.423132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422639    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.423132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422652    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e197b26-0b5d-40c3-b4ff-7f120cec07d9-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.423132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.422818    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:12.424040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.424002    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:12.424386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.424344    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:00:12.424544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.424524    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "kube-api-access-f5m6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:12.425175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.425145    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy" pod "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" (UID: "1c196c76-fa6a-49d0-8c5f-d9f1959f75e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:12.523949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.523909    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.523949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.523948    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.524259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.523963    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5m6j\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-kube-api-access-f5m6j\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.524259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.523979    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.524259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.523992    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.524259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.524005    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:12.626937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.626901    4515 generic.go:356] "Generic (PLEG): container finished" podID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerID="296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572" exitCode=0
Nov 02 08:00:12.627125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.626964    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"
Nov 02 08:00:12.627125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.626978    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" event={"ID":"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0","Type":"ContainerDied","Data":"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"}
Nov 02 08:00:12.627125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.627065    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4" event={"ID":"1c196c76-fa6a-49d0-8c5f-d9f1959f75e0","Type":"ContainerDied","Data":"16410efd5421dc654e08856206b2118bfefc36a4044e73741d001b4c35eb7bc9"}
Nov 02 08:00:12.627125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.627098    4515 scope.go:122] "RemoveContainer" containerID="296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"
Nov 02 08:00:12.628582 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.628552    4515 generic.go:356] "Generic (PLEG): container finished" podID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerID="26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196" exitCode=0
Nov 02 08:00:12.628724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.628577    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" event={"ID":"1e197b26-0b5d-40c3-b4ff-7f120cec07d9","Type":"ContainerDied","Data":"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"}
Nov 02 08:00:12.628724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.628627    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"
Nov 02 08:00:12.628724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.628627    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz" event={"ID":"1e197b26-0b5d-40c3-b4ff-7f120cec07d9","Type":"ContainerDied","Data":"1196f46a786827984594d2ec7724546f328c8041e9854b26f11aee011f81f3d7"}
Nov 02 08:00:12.639724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.639709    4515 scope.go:122] "RemoveContainer" containerID="296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"
Nov 02 08:00:12.639978 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:12.639955    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572\": container with ID starting with 296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572 not found: ID does not exist" containerID="296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"
Nov 02 08:00:12.640082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.639983    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572"} err="failed to get container status \"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572\": rpc error: code = NotFound desc = could not find container \"296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572\": container with ID starting with 296be741e8b5d909edf2a033ca1aacb511245af79bc2a01e95ca1f90de23d572 not found: ID does not exist"
Nov 02 08:00:12.640082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.640000    4515 scope.go:122] "RemoveContainer" containerID="26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"
Nov 02 08:00:12.649242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.649220    4515 scope.go:122] "RemoveContainer" containerID="26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"
Nov 02 08:00:12.650244 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:12.650227    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196\": container with ID starting with 26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196 not found: ID does not exist" containerID="26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"
Nov 02 08:00:12.650315 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.650251    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196"} err="failed to get container status \"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196\": rpc error: code = NotFound desc = could not find container \"26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196\": container with ID starting with 26a8910dad6884c27c1272b3d60f0bfc997e3a42dedc9978d17502b761acb196 not found: ID does not exist"
Nov 02 08:00:12.684871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.684839    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 08:00:12.696589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.696563    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-7f38f1-openshift-default-59c89b95-f78n4"]
Nov 02 08:00:12.722012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.721979    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 08:00:12.728710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:12.728685    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8fmqz"]
Nov 02 08:00:13.374507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:13.374475    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" path="/var/lib/kubelet/pods/1c196c76-fa6a-49d0-8c5f-d9f1959f75e0/volumes"
Nov 02 08:00:13.375021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:13.375006    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" path="/var/lib/kubelet/pods/1e197b26-0b5d-40c3-b4ff-7f120cec07d9/volumes"
Nov 02 08:00:16.366685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:16.366656    4515 scope.go:122] "RemoveContainer" containerID="028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f"
Nov 02 08:00:16.660990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:16.660958    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerStarted","Data":"2be07ca115fff2b3b9ae524ed929a65d1c96d8d7dcb53397d8a58f602bc0bad3"}
Nov 02 08:00:17.313181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.313140    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:00:17.314195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314175    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerName="istio-proxy"
Nov 02 08:00:17.314267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314197    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerName="istio-proxy"
Nov 02 08:00:17.314267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314217    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy"
Nov 02 08:00:17.314267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314225    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy"
Nov 02 08:00:17.314378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314274    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy"
Nov 02 08:00:17.314378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314281    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy"
Nov 02 08:00:17.314447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314435    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4bba617-9ead-4c0a-9bb6-689b8c16de59" containerName="istio-proxy"
Nov 02 08:00:17.314480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314460    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e197b26-0b5d-40c3-b4ff-7f120cec07d9" containerName="istio-proxy"
Nov 02 08:00:17.314480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.314474    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c196c76-fa6a-49d0-8c5f-d9f1959f75e0" containerName="istio-proxy"
Nov 02 08:00:17.318633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.318608    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.344119 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.344087    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-736f7c-openshift-default-dockercfg-5m6sv\""
Nov 02 08:00:17.363935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.363902    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429196    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429271    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429363    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429410    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qvv\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429484    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429535    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429646    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429676    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.429757    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.442381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.438563    4515 scope.go:122] "RemoveContainer" containerID="e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"
Nov 02 08:00:17.569587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569555    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.569724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569600    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.569724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569686    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.569847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569798    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.569913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569844    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.569913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569889    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.570018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.569937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qvv\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.570070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.570017    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.570125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.570076    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.570953 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.570876    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.571606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.571583    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.572219 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.572199    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.572648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.572626    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.573004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.572977    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.583140 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.583105    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.585217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.585196    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.599104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.599085    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.611376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.610814    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qvv\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\") pod \"e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") " pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.634241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.634211    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:17.676361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.673726    4515 generic.go:356] "Generic (PLEG): container finished" podID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerID="2be07ca115fff2b3b9ae524ed929a65d1c96d8d7dcb53397d8a58f602bc0bad3" exitCode=60
Nov 02 08:00:17.676361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.673794    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerDied","Data":"2be07ca115fff2b3b9ae524ed929a65d1c96d8d7dcb53397d8a58f602bc0bad3"}
Nov 02 08:00:17.676361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.673837    4515 scope.go:122] "RemoveContainer" containerID="028fec1ff56a0a950fa3cb39f4cc219baae557776abb1f5f1e441a2acfac549f"
Nov 02 08:00:17.676361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.674252    4515 scope.go:122] "RemoveContainer" containerID="2be07ca115fff2b3b9ae524ed929a65d1c96d8d7dcb53397d8a58f602bc0bad3"
Nov 02 08:00:17.676361 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:17.674621    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd_odh-ai-gateway-infra(bb3ba74f-eb5d-411d-b8a3-be421ebf6206)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206"
Nov 02 08:00:17.731898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:17.731654    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:17.740296 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:17.738985    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:00:18.001952 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:18.001921    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a06d99_ea63_49a8_ab4d_e26185d54896.slice/crio-a760614990b3e09d45d7302771920b8bd9fcd621833d16014fced6ce76a351fa WatchSource:0}: Error finding container a760614990b3e09d45d7302771920b8bd9fcd621833d16014fced6ce76a351fa: Status 404 returned error can't find the container with id a760614990b3e09d45d7302771920b8bd9fcd621833d16014fced6ce76a351fa
Nov 02 08:00:18.003292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.003268    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:00:18.005091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.005056    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:18.005177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.005125    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:18.005177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.005158    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:18.769361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.767401    4515 generic.go:356] "Generic (PLEG): container finished" podID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerID="b278d175947645df3157e9501ba15eaa1bf55704a051a39580e598a10d171a31" exitCode=60
Nov 02 08:00:18.769361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.767647    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerDied","Data":"b278d175947645df3157e9501ba15eaa1bf55704a051a39580e598a10d171a31"}
Nov 02 08:00:18.769361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.767725    4515 scope.go:122] "RemoveContainer" containerID="e43839a35c182f6d1ede831a722c48210fcc242b9bc8e61fad44cb6db5da05ba"
Nov 02 08:00:18.769361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.768445    4515 scope.go:122] "RemoveContainer" containerID="b278d175947645df3157e9501ba15eaa1bf55704a051a39580e598a10d171a31"
Nov 02 08:00:18.769361 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:18.768946    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf_odh-ai-gateway-infra(d78b0c0f-59d2-4ddf-abb7-8652d59d3102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102"
Nov 02 08:00:18.781452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.778944    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerStarted","Data":"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"}
Nov 02 08:00:18.781452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.779021    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerStarted","Data":"a760614990b3e09d45d7302771920b8bd9fcd621833d16014fced6ce76a351fa"}
Nov 02 08:00:18.931834 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.931785    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podStartSLOduration=1.931769976 podStartE2EDuration="1.931769976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:18.930605426 +0000 UTC m=+2163.693042331" watchObservedRunningTime="2026-11-02 08:00:18.931769976 +0000 UTC m=+2163.694206882"
Nov 02 08:00:18.945636 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:18.945613    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:19.014843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.014798    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\") pod \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") "
Nov 02 08:00:19.015195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.015169    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\" (UniqueName: \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\") pod \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\" (UID: \"bb3ba74f-eb5d-411d-b8a3-be421ebf6206\") "
Nov 02 08:00:19.015382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.015355    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca" pod "bb3ba74f-eb5d-411d-b8a3-be421ebf6206" (UID: "bb3ba74f-eb5d-411d-b8a3-be421ebf6206"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:19.015903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.015849    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.018721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.018694    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb" pod "bb3ba74f-eb5d-411d-b8a3-be421ebf6206" (UID: "bb3ba74f-eb5d-411d-b8a3-be421ebf6206"). InnerVolumeSpecName "kube-api-access-zlpfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:19.116980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.116935    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpfb\" (UniqueName: \"kubernetes.io/projected/bb3ba74f-eb5d-411d-b8a3-be421ebf6206-kube-api-access-zlpfb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.375285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.375248    4515 scope.go:122] "RemoveContainer" containerID="fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"
Nov 02 08:00:19.634936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.634908    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:19.641094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.641066    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:19.791381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.791301    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerStarted","Data":"36351790c2bb25b16b1acf3dbcf542f71cd9905b55f386d1abb1a29e06b87d6f"}
Nov 02 08:00:19.793236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.793208    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"
Nov 02 08:00:19.793236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.793222    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd" event={"ID":"bb3ba74f-eb5d-411d-b8a3-be421ebf6206","Type":"ContainerDied","Data":"7de9f378a10a9c38406e6b72334ab61e7630448f55081e8c5d8b21d9a8c3e15a"}
Nov 02 08:00:19.793493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.793289    4515 scope.go:122] "RemoveContainer" containerID="2be07ca115fff2b3b9ae524ed929a65d1c96d8d7dcb53397d8a58f602bc0bad3"
Nov 02 08:00:19.793565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.793518    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:19.795337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.795131    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:00:19.844840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.844800    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:19.884108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.884022    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:19.887867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:19.887834    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893440-wfmxd"]
Nov 02 08:00:20.026175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.026149    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:20.129933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.129896    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\") pod \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") "
Nov 02 08:00:20.130244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.130222    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\" (UniqueName: \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\") pod \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\" (UID: \"d78b0c0f-59d2-4ddf-abb7-8652d59d3102\") "
Nov 02 08:00:20.130329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.130243    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca" pod "d78b0c0f-59d2-4ddf-abb7-8652d59d3102" (UID: "d78b0c0f-59d2-4ddf-abb7-8652d59d3102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:20.130842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.130822    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:20.132464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.132439    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5" pod "d78b0c0f-59d2-4ddf-abb7-8652d59d3102" (UID: "d78b0c0f-59d2-4ddf-abb7-8652d59d3102"). InnerVolumeSpecName "kube-api-access-zclr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:20.231936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.231903    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zclr5\" (UniqueName: \"kubernetes.io/projected/d78b0c0f-59d2-4ddf-abb7-8652d59d3102-kube-api-access-zclr5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:20.810156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.808468    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"
Nov 02 08:00:20.810156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.808478    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf" event={"ID":"d78b0c0f-59d2-4ddf-abb7-8652d59d3102","Type":"ContainerDied","Data":"c99d975a084eac23838281d4411f2fd5f96ad1814e0f65228c242e9a75acba01"}
Nov 02 08:00:20.810156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.808544    4515 scope.go:122] "RemoveContainer" containerID="b278d175947645df3157e9501ba15eaa1bf55704a051a39580e598a10d171a31"
Nov 02 08:00:20.812555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.812521    4515 generic.go:356] "Generic (PLEG): container finished" podID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerID="36351790c2bb25b16b1acf3dbcf542f71cd9905b55f386d1abb1a29e06b87d6f" exitCode=60
Nov 02 08:00:20.812682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.812603    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerDied","Data":"36351790c2bb25b16b1acf3dbcf542f71cd9905b55f386d1abb1a29e06b87d6f"}
Nov 02 08:00:20.813770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.813746    4515 scope.go:122] "RemoveContainer" containerID="36351790c2bb25b16b1acf3dbcf542f71cd9905b55f386d1abb1a29e06b87d6f"
Nov 02 08:00:20.814279 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:20.814121    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz_odh-ai-gateway-infra(b7c041a2-7d09-4166-b11e-b0036340cb8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d"
Nov 02 08:00:20.821047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.821031    4515 scope.go:122] "RemoveContainer" containerID="fed351d3e6dd54c05671a31a5903a73d87be924295e3d60518bd7490ffb32ace"
Nov 02 08:00:20.861887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.861856    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:20.889257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.889223    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:20.908839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:20.908804    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893440-g7xsf"]
Nov 02 08:00:21.378902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:21.378867    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" path="/var/lib/kubelet/pods/bb3ba74f-eb5d-411d-b8a3-be421ebf6206/volumes"
Nov 02 08:00:21.379421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:21.379406    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" path="/var/lib/kubelet/pods/d78b0c0f-59d2-4ddf-abb7-8652d59d3102/volumes"
Nov 02 08:00:22.032361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.032334    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:22.062952 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.062920    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\" (UniqueName: \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\") pod \"b7c041a2-7d09-4166-b11e-b0036340cb8d\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") "
Nov 02 08:00:22.063114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.063062    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\") pod \"b7c041a2-7d09-4166-b11e-b0036340cb8d\" (UID: \"b7c041a2-7d09-4166-b11e-b0036340cb8d\") "
Nov 02 08:00:22.063460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.063438    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca" pod "b7c041a2-7d09-4166-b11e-b0036340cb8d" (UID: "b7c041a2-7d09-4166-b11e-b0036340cb8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:22.065176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.065152    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv" pod "b7c041a2-7d09-4166-b11e-b0036340cb8d" (UID: "b7c041a2-7d09-4166-b11e-b0036340cb8d"). InnerVolumeSpecName "kube-api-access-l9nsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:22.164793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.164754    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c041a2-7d09-4166-b11e-b0036340cb8d-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:22.164793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.164787    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nsv\" (UniqueName: \"kubernetes.io/projected/b7c041a2-7d09-4166-b11e-b0036340cb8d-kube-api-access-l9nsv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:22.832030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.831998    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"
Nov 02 08:00:22.832231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.832026    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz" event={"ID":"b7c041a2-7d09-4166-b11e-b0036340cb8d","Type":"ContainerDied","Data":"d37eda962d49607fdfbb11ca7b6c523dd2f0b790e73d36b94af2029cfb1936a2"}
Nov 02 08:00:22.832231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.832090    4515 scope.go:122] "RemoveContainer" containerID="36351790c2bb25b16b1acf3dbcf542f71cd9905b55f386d1abb1a29e06b87d6f"
Nov 02 08:00:22.888191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.888158    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:22.891377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.891352    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893440-kvvpz"]
Nov 02 08:00:22.980637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:22.980586    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerName="main" containerID="cri-o://3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0" gracePeriod=45
Nov 02 08:00:23.121778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.121743    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerName="main" probeResult="failure" output="Get \"https://10.133.0.205:8000/ready\": dial tcp 10.133.0.205:8000: connect: connection refused"
Nov 02 08:00:23.270206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.270173    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 08:00:23.354020 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.353932    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354020 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354000    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354428    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354567    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354570    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:23.354629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354604    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354649    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\" (UniqueName: \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\") pod \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\" (UID: \"e5f162cd-3ba1-43a6-ad53-622c47e7e392\") "
Nov 02 08:00:23.354806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.354783    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:23.355619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.355586    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.355730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.355621    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.358057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.356649    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:23.358057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.356901    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:23.360737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.360709    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:00:23.361202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.361177    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp" pod "e5f162cd-3ba1-43a6-ad53-622c47e7e392" (UID: "e5f162cd-3ba1-43a6-ad53-622c47e7e392"). InnerVolumeSpecName "kube-api-access-bq5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:23.378986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.378952    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" path="/var/lib/kubelet/pods/b7c041a2-7d09-4166-b11e-b0036340cb8d/volumes"
Nov 02 08:00:23.456931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.456897    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.456931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.456925    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.456931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.456936    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f162cd-3ba1-43a6-ad53-622c47e7e392-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.457181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.456945    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5rp\" (UniqueName: \"kubernetes.io/projected/e5f162cd-3ba1-43a6-ad53-622c47e7e392-kube-api-access-bq5rp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.860358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.852362    4515 generic.go:356] "Generic (PLEG): container finished" podID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerID="3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0" exitCode=0
Nov 02 08:00:23.860358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.852533    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" event={"ID":"e5f162cd-3ba1-43a6-ad53-622c47e7e392","Type":"ContainerDied","Data":"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"}
Nov 02 08:00:23.860358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.852568    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh" event={"ID":"e5f162cd-3ba1-43a6-ad53-622c47e7e392","Type":"ContainerDied","Data":"c105b318ed7d5d5d375e5dbcc620a17e14ef504703289ec7ac6215c6ffd588a0"}
Nov 02 08:00:23.860358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.852591    4515 scope.go:122] "RemoveContainer" containerID="3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"
Nov 02 08:00:23.860358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.852750    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"
Nov 02 08:00:23.894938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.894910    4515 scope.go:122] "RemoveContainer" containerID="3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"
Nov 02 08:00:23.895391 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:23.895295    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0\": container with ID starting with 3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0 not found: ID does not exist" containerID="3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"
Nov 02 08:00:23.895495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.895397    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0"} err="failed to get container status \"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0\": rpc error: code = NotFound desc = could not find container \"3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0\": container with ID starting with 3e9b6fd3a94b30d1391c4271a440b37dacdf9787287dddf6a763587b5af778c0 not found: ID does not exist"
Nov 02 08:00:23.919763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.919679    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 08:00:23.931794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:23.931761    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e4c131/auth-test-model-e4c131-kserve-787d958df7-vrzsh"]
Nov 02 08:00:25.374680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.374644    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" path="/var/lib/kubelet/pods/e5f162cd-3ba1-43a6-ad53-622c47e7e392/volumes"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.587935    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.588938    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerName="main"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.588961    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerName="main"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.588981    4515 cpu_manager.go:414] "containerMap: removing container" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.588990    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589003    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589012    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589063    4515 cpu_manager.go:414] "containerMap: removing container" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589070    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589093    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589100    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589123    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589130    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589147    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.589198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589154    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589387    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589410    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589424    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589435    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589444    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589453    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589467    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589478    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f162cd-3ba1-43a6-ad53-622c47e7e392" containerName="main"
Nov 02 08:00:25.590187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.589488    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:25.609352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.604592    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.655324 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.654462    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7a8bb6-openshift-default-dockercfg-8lb7h\""
Nov 02 08:00:25.664732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.664070    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:00:25.700435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700404    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700443    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700543    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700581    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700639    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700709    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700801    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lmk\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.700862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.701025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.700866    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802136    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lmk\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802189    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802221    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802269    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802288    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802355    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802382    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802451    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802483    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802693    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802784    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.802937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802860    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.803113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.802970    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.803175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.803153    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.804727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.804703    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.804961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.804942    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.878933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.862059    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lmk\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.913984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.913874    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:25.956262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:25.956226    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:26.290894 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:26.290863    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace90e6c_6b77_4cd6_9a5d_d4355434a67e.slice/crio-e5e8617b36fd16ba714f63c92c3ff1f6442f65c9b3b9193e5917693fe0da28c4 WatchSource:0}: Error finding container e5e8617b36fd16ba714f63c92c3ff1f6442f65c9b3b9193e5917693fe0da28c4: Status 404 returned error can't find the container with id e5e8617b36fd16ba714f63c92c3ff1f6442f65c9b3b9193e5917693fe0da28c4
Nov 02 08:00:26.291872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.291847    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:00:26.293741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.293704    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:26.293858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.293782    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:26.293858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.293814    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:26.932589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.932556    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerStarted","Data":"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"}
Nov 02 08:00:26.933115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.932608    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerStarted","Data":"e5e8617b36fd16ba714f63c92c3ff1f6442f65c9b3b9193e5917693fe0da28c4"}
Nov 02 08:00:26.956451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.956416    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:26.993195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:26.993145    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podStartSLOduration=1.993131312 podStartE2EDuration="1.993131312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:26.989840225 +0000 UTC m=+2171.752277131" watchObservedRunningTime="2026-11-02 08:00:26.993131312 +0000 UTC m=+2171.755568215"
Nov 02 08:00:27.963729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:27.963697    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:28.946049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:28.946016    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:28.947241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:28.947212    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:00:33.911332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:33.910161    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:00:47.724691 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:00:47.724645    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:00:52.546881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.546843    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:00:52.548171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548142    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:52.548171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548164    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78b0c0f-59d2-4ddf-abb7-8652d59d3102" containerName="cleanup"
Nov 02 08:00:52.548442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548220    4515 cpu_manager.go:414] "containerMap: removing container" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:52.548442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548228    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:52.548442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548243    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:52.548442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548266    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c041a2-7d09-4166-b11e-b0036340cb8d" containerName="cleanup"
Nov 02 08:00:52.548650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.548462    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3ba74f-eb5d-411d-b8a3-be421ebf6206" containerName="cleanup"
Nov 02 08:00:52.552383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.552360    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.583190 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.583118    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:00:52.587906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.587880    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ea9556-openshift-default-dockercfg-l6wxt\""
Nov 02 08:00:52.608154 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608113    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608402 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608177    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608402 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608205    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qjp\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608402 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608246    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608460    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608515    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608617    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608655    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.608708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.608689    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710433    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710473    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710573    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710637    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710664    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qjp\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710712    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.710921    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.714005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.711512    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.726229 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.726107    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.728015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.726824    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.728015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.727126    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.728015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.727427    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.730268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.729133    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.743632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.743556    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.755783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.755755    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qjp\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.757816 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.757790    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\") pod \"e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") " pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:52.870667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:52.870576    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:53.222663 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:00:53.222619    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4ed0f8_7aea_458b_b071_fe76cfbbe123.slice/crio-85d95c5d99cad914ccb59d19856893fd88ea9742f2a2a9866f50c500188c8c24 WatchSource:0}: Error finding container 85d95c5d99cad914ccb59d19856893fd88ea9742f2a2a9866f50c500188c8c24: Status 404 returned error can't find the container with id 85d95c5d99cad914ccb59d19856893fd88ea9742f2a2a9866f50c500188c8c24
Nov 02 08:00:53.222663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:53.222643    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:00:53.225531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:53.225500    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:53.225627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:53.225574    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:53.225627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:53.225609    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:54.179405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:54.179369    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerStarted","Data":"bcebaf2f4ce7045167bf1ac58c9c85bdf9c63da0a5424e1beca9f71aa1031b12"}
Nov 02 08:00:54.179843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:54.179427    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerStarted","Data":"85d95c5d99cad914ccb59d19856893fd88ea9742f2a2a9866f50c500188c8c24"}
Nov 02 08:00:54.246084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:54.246029    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podStartSLOduration=2.246015091 podStartE2EDuration="2.246015091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:54.245634975 +0000 UTC m=+2199.008071894" watchObservedRunningTime="2026-11-02 08:00:54.246015091 +0000 UTC m=+2199.008451997"
Nov 02 08:00:54.870942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:54.870906    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:54.876919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:54.876891    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:55.186682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:55.186603    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:55.187528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:55.187510    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:00:58.835516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.834498    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:00:58.882328 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.881949    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:00:58.882328 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.882087    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:58.920087 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.919934    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:00:58.920087 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.920024    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ea9556-dockercfg-jgxbp\""
Nov 02 08:00:58.929933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.929891    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7hd\" (UniqueName: \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:58.930300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:58.930274    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.031555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.031523    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7hd\" (UniqueName: \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.031764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.031625    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.032478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.032453    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.055920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.055887    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7hd\" (UniqueName: \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\") pod \"payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.211789 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.211707    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:00:59.523549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:00:59.523518    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:01:00.220800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:00.220763    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" event={"ID":"6f3a7eb8-ec63-4a01-91b6-caa289e632dd","Type":"ContainerStarted","Data":"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"}
Nov 02 08:01:00.221244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:00.220822    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" event={"ID":"6f3a7eb8-ec63-4a01-91b6-caa289e632dd","Type":"ContainerStarted","Data":"667c9e76146fa62a305ae2cdc93fe7a6d23c6b73624c6395e9983358ba82fe9c"}
Nov 02 08:01:00.221244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:00.220854    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:01:11.231975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:11.231940    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:01:11.274153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:11.274065    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" podStartSLOduration=13.274050171 podStartE2EDuration="13.274050171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:00.281261691 +0000 UTC m=+2205.043698599" watchObservedRunningTime="2026-11-02 08:01:11.274050171 +0000 UTC m=+2216.036487078"
Nov 02 08:01:14.376286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.376194    4515 generic.go:356] "Generic (PLEG): container finished" podID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerID="0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d" exitCode=0
Nov 02 08:01:14.376286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.376239    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerDied","Data":"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"}
Nov 02 08:01:14.376799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.376620    4515 scope.go:122] "RemoveContainer" containerID="0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"
Nov 02 08:01:14.378268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.378235    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:14.378356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.378339    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:14.378412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:14.378374    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:15.388167 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:15.388130    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerStarted","Data":"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"}
Nov 02 08:01:15.641395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:15.640376    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:01:15.642610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:15.642582    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 08:01:15.642774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:15.642645    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 08:01:16.639333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:16.638681    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 08:01:16.639333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:16.638760    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 08:01:17.638334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:17.638209    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 08:01:17.638334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:17.638267    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 08:01:17.641975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:17.641946    4515 scope.go:122] "RemoveContainer" containerID="aa450f58cbd1da716c6c99bfddd15d0fb660b3c21ca24c249023c92f7b3f6919"
Nov 02 08:01:17.747635 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:01:17.747595    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:01:18.639332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:18.638712    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 08:01:18.639332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:18.638775    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 08:01:19.635737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:19.635699    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 08:01:19.636367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:19.636334    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 08:01:19.794170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:19.794136    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:01:20.640247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:20.640216    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:01:20.641412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:20.641386    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:01:27.945332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:27.942320    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:01:39.580110 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.580067    4515 generic.go:356] "Generic (PLEG): container finished" podID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerID="1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d" exitCode=0
Nov 02 08:01:39.580593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.580143    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerDied","Data":"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"}
Nov 02 08:01:39.580664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.580645    4515 scope.go:122] "RemoveContainer" containerID="1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"
Nov 02 08:01:39.582665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.582630    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:39.582768 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.582707    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:39.582768 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:39.582747    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:40.632696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:40.632656    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerStarted","Data":"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"}
Nov 02 08:01:40.956646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:40.956543    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:01:40.958461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:40.958421    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:40.958627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:40.958491    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:41.959251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:41.959211    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:41.959913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:41.959878    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:42.960887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:42.960832    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:42.961540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:42.961506    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:43.946237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:43.946203    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:01:43.957006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:43.956976    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:43.957174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:43.957024    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:44.957412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:44.957373    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:44.957838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:44.957467    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:45.957656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:45.957621    4515 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 08:01:45.958113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:45.957677    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 08:01:46.699454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.699412    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:01:46.704520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.704497    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.722063 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.722030    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"agc-cred-2a9bfe236817\""
Nov 02 08:01:46.722260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.722206    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"payload-processing-external-model-e2e--634ed34c-dockercfg-lmlct\""
Nov 02 08:01:46.736358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.735030    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"kube-root-ca.crt\""
Nov 02 08:01:46.736358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.735206    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bb4907\""
Nov 02 08:01:46.736358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.735879    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"openshift-service-ca.crt\""
Nov 02 08:01:46.736358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.735967    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"routing-overlay\""
Nov 02 08:01:46.740893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.740861    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:01:46.763271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.763235    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.763546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.763493    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.763692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.763671    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.763766 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.763712    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpv6\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.875453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.875353    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.877089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.877052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.883562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.883536    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.884252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.884229    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpv6\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.884901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.876701    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.885053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.878067    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.888877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.888832    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:46.915342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:46.915259    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpv6\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:47.019721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.019686    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:47.120809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.120733    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:01:47.125179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.123542    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:01:47.336052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.335018    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:01:47.603513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.603477    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:01:47.611627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.611597    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.626602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.626575    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:01:47.626982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.626957    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2dde84-dockercfg-nncnp\""
Nov 02 08:01:47.631086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.630343    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:01:47.715912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.699059    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnck4\" (UniqueName: \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.715912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.699140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.715912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.710787    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:01:47.786473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.735439    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:47.786473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.755642    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:01:47.810356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.807957    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnck4\" (UniqueName: \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.810356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.808032    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.810356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.809205    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.878384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.867013    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnck4\" (UniqueName: \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\") pod \"payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.878384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.867497    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" event={"ID":"5a81c065-9080-4fa6-89f7-06b0a14749e8","Type":"ContainerStarted","Data":"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"}
Nov 02 08:01:47.878384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.867550    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" event={"ID":"5a81c065-9080-4fa6-89f7-06b0a14749e8","Type":"ContainerStarted","Data":"cd16d27173935604fdc0680208bdba8382fba79bddaf8d113488a8742000b3bb"}
Nov 02 08:01:47.878384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.867595    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:47.932056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.930835    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:47.950680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.940698    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtcx\" (UniqueName: \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:47.950680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:47.940890    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:47.977166 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:01:47.976838    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:01:48.042420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.042389    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtcx\" (UniqueName: \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.042764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.042496    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.043598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.043574    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.067782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.067752    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtcx\" (UniqueName: \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\") pod \"payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.169700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.169611    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.281240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.281191    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" podStartSLOduration=2.28117244 podStartE2EDuration="2.28117244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:47.96759539 +0000 UTC m=+2252.730032321" watchObservedRunningTime="2026-11-02 08:01:48.28117244 +0000 UTC m=+2253.043609364"
Nov 02 08:01:48.282034 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:01:48.281998    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd607758e_3dfe_4698_95d2_c79123967559.slice/crio-69d24a2bc89423c89f0ff21611e17de6c3a490ced943dda7e6b791c46009c49a WatchSource:0}: Error finding container 69d24a2bc89423c89f0ff21611e17de6c3a490ced943dda7e6b791c46009c49a: Status 404 returned error can't find the container with id 69d24a2bc89423c89f0ff21611e17de6c3a490ced943dda7e6b791c46009c49a
Nov 02 08:01:48.283088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.283067    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:01:48.496008 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:01:48.495977    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9670062_f9c8_4486_87db_207a93997f16.slice/crio-8b22421928a3045ac2263acc7bc7f5d097a8101cfc1704906312bbac016f8502 WatchSource:0}: Error finding container 8b22421928a3045ac2263acc7bc7f5d097a8101cfc1704906312bbac016f8502: Status 404 returned error can't find the container with id 8b22421928a3045ac2263acc7bc7f5d097a8101cfc1704906312bbac016f8502
Nov 02 08:01:48.497144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.497113    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:01:48.876120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.876084    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" event={"ID":"d607758e-3dfe-4698-95d2-c79123967559","Type":"ContainerStarted","Data":"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"}
Nov 02 08:01:48.876341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.876136    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" event={"ID":"d607758e-3dfe-4698-95d2-c79123967559","Type":"ContainerStarted","Data":"69d24a2bc89423c89f0ff21611e17de6c3a490ced943dda7e6b791c46009c49a"}
Nov 02 08:01:48.876341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.876160    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:01:48.877860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.877830    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" event={"ID":"b9670062-f9c8-4486-87db-207a93997f16","Type":"ContainerStarted","Data":"e65a75946d6fa579469a2fb2fd0930ae012281930df25bd39397201da5bff096"}
Nov 02 08:01:48.877988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.877876    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" event={"ID":"b9670062-f9c8-4486-87db-207a93997f16","Type":"ContainerStarted","Data":"8b22421928a3045ac2263acc7bc7f5d097a8101cfc1704906312bbac016f8502"}
Nov 02 08:01:48.878064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.878047    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:48.918942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.918893    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" podStartSLOduration=1.9188803060000001 podStartE2EDuration="1.918880306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:48.917515135 +0000 UTC m=+2253.679952040" watchObservedRunningTime="2026-11-02 08:01:48.918880306 +0000 UTC m=+2253.681317212"
Nov 02 08:01:48.962523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:48.962473    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" podStartSLOduration=1.962458566 podStartE2EDuration="1.962458566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:48.958802874 +0000 UTC m=+2253.721239781" watchObservedRunningTime="2026-11-02 08:01:48.962458566 +0000 UTC m=+2253.724895471"
Nov 02 08:01:58.882551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:58.882517    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:01:59.887621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:59.887585    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:01:59.888020 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:01:59.887660    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:02:02.083081 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.083040    4515 generic.go:356] "Generic (PLEG): container finished" podID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerID="bcebaf2f4ce7045167bf1ac58c9c85bdf9c63da0a5424e1beca9f71aa1031b12" exitCode=0
Nov 02 08:02:02.083743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.083172    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerDied","Data":"bcebaf2f4ce7045167bf1ac58c9c85bdf9c63da0a5424e1beca9f71aa1031b12"}
Nov 02 08:02:02.083743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.083727    4515 scope.go:122] "RemoveContainer" containerID="bcebaf2f4ce7045167bf1ac58c9c85bdf9c63da0a5424e1beca9f71aa1031b12"
Nov 02 08:02:02.105851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.105362    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:02.105851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.105474    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:02.105851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:02.105521    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:03.093949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:03.093910    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerStarted","Data":"83519c6bdcbf45817cd4091cb77a010b29ab9af27eb3c5b7e008acc413d1423c"}
Nov 02 08:02:03.871015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:03.870981    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:02:03.871264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:03.871241    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 08:02:03.871359 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:03.871288    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 08:02:04.871894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:04.871857    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 08:02:04.872347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:04.871912    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 08:02:05.871817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:05.871777    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 08:02:05.872027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:05.871840    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 08:02:06.871322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:06.871269    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 08:02:06.871541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:06.871345    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 08:02:07.875266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:07.874716    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 08:02:07.875266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:07.874788    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 08:02:08.891195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:08.891163    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:02:08.896016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:08.894714    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:02:08.896205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:08.896080    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:02:17.724812 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:02:17.724775    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:02:19.965865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:19.965829    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:02:20.109456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.109423    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:02:20.115424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.115398    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.135092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.135065    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-71a41c-openshift-default-dockercfg-5w98s\""
Nov 02 08:02:20.150985 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.150956    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151021    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151164    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151194    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151221    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151319    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6q5\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151402 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151345    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151414    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.151553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.151535    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.161510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.161477    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266233    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266294    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266343    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266388    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6q5\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266415    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266489    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266641    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.267332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.266778    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.268009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.267751    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.291333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.269056    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.291333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.273160    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.291333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.273457    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.291333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.277983    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.294477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.293842    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.319535 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.319261    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.322130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.321912    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6q5\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.409332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.397954    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\") pod \"e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") " pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.432961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.432928    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:20.813462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.812689    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:02:20.815488 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:02:20.815456    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1dffec_838b_4f8c_b690_23fea9d4e87c.slice/crio-444fc7e33d692fec93dff80c0e755c88c9f005baaf75d4a9042c2f919c96086e WatchSource:0}: Error finding container 444fc7e33d692fec93dff80c0e755c88c9f005baaf75d4a9042c2f919c96086e: Status 404 returned error can't find the container with id 444fc7e33d692fec93dff80c0e755c88c9f005baaf75d4a9042c2f919c96086e
Nov 02 08:02:20.818702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.818665    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:20.818825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.818751    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:20.818825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:20.818802    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:21.346010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.345974    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerStarted","Data":"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"}
Nov 02 08:02:21.346561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.346043    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerStarted","Data":"444fc7e33d692fec93dff80c0e755c88c9f005baaf75d4a9042c2f919c96086e"}
Nov 02 08:02:21.393453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.393393    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podStartSLOduration=1.393376266 podStartE2EDuration="1.393376266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:21.391003116 +0000 UTC m=+2286.153440049" watchObservedRunningTime="2026-11-02 08:02:21.393376266 +0000 UTC m=+2286.155813219"
Nov 02 08:02:21.433377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.433341    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:21.435026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.434996    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:02:21.435186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:21.435053    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:02:22.438570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:22.438536    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:23.378381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:23.378351    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:23.378572 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:23.378418    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:02:25.550083 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:25.550039    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 08:02:25.550604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:25.550287    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy" containerID="cri-o://bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984" gracePeriod=30
Nov 02 08:02:30.822728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.822665    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 08:02:30.913284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913241    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913298    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913376    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913423    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913614    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913738    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:30.913809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913744    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.913932 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913910    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:30.913997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913916    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:02:30.913997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.913948    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.914089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914027    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.914089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914067    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\") pod \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\" (UID: \"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77\") "
Nov 02 08:02:30.914241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914114    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:30.914346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914293    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:30.914845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914766    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:30.914845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914789    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:30.914845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914801    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:30.914845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914812    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:30.914845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.914823    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:30.916267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.916224    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:02:30.916453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.916430    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "kube-api-access-f4w2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:30.916624 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.916603    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:30.916724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:30.916707    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token" pod "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" (UID: "1b3e20a5-bc1d-406a-9fe0-03ac34c66c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:31.016487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.016455    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:31.016487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.016484    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:31.016487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.016495    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:31.016733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.016504    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4w2n\" (UniqueName: \"kubernetes.io/projected/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77-kube-api-access-f4w2n\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:02:31.430835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.430797    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerID="bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984" exitCode=0
Nov 02 08:02:31.431018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.430846    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"
Nov 02 08:02:31.431018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.430874    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerDied","Data":"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"}
Nov 02 08:02:31.431018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.430942    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h" event={"ID":"1b3e20a5-bc1d-406a-9fe0-03ac34c66c77","Type":"ContainerDied","Data":"625beb8c87792e125f9c250ecf2668e5388caab2d98b3d1e547274310cad1260"}
Nov 02 08:02:31.431018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.430973    4515 scope.go:122] "RemoveContainer" containerID="bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"
Nov 02 08:02:31.441949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.441930    4515 scope.go:122] "RemoveContainer" containerID="2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"
Nov 02 08:02:31.455089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.455067    4515 scope.go:122] "RemoveContainer" containerID="bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"
Nov 02 08:02:31.455440 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:02:31.455416    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984\": container with ID starting with bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984 not found: ID does not exist" containerID="bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"
Nov 02 08:02:31.455534 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.455448    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984"} err="failed to get container status \"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984\": rpc error: code = NotFound desc = could not find container \"bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984\": container with ID starting with bf0f344d02cc6c7b115cf27b6160f832ead96145ffbf1b4b26e00648e20c8984 not found: ID does not exist"
Nov 02 08:02:31.455534 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.455467    4515 scope.go:122] "RemoveContainer" containerID="2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"
Nov 02 08:02:31.455755 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:02:31.455734    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a\": container with ID starting with 2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a not found: ID does not exist" containerID="2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"
Nov 02 08:02:31.455809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.455762    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a"} err="failed to get container status \"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a\": rpc error: code = NotFound desc = could not find container \"2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a\": container with ID starting with 2ab96b76a44f35af8be1325a20c132fbe1cfbc8db4b287e5212a7826c7af2e4a not found: ID does not exist"
Nov 02 08:02:31.500326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.500271    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 08:02:31.510107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:31.510076    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bph7h"]
Nov 02 08:02:33.399602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:02:33.399566    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" path="/var/lib/kubelet/pods/1b3e20a5-bc1d-406a-9fe0-03ac34c66c77/volumes"
Nov 02 08:02:47.724984 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:02:47.724948    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:03:13.602355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:13.599536    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:03:17.727242 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:03:17.727166    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:03:19.938827 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:19.938782    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:03:19.940000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:19.939946    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy" containerID="cri-o://ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb" gracePeriod=30
Nov 02 08:03:22.549692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.549644    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:03:22.551197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551168    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.551197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551194    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.551412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551223    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.551412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551230    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.551412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551369    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.551412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.551392    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3e20a5-bc1d-406a-9fe0-03ac34c66c77" containerName="istio-proxy"
Nov 02 08:03:22.557282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.557260    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.566330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.566272    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:03:22.576192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.575105    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"agc-cred-f32e6bb1571b\""
Nov 02 08:03:22.719848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.719803    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.720045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.719883    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.720132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.720115    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.720267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.720243    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmgd\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.821858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.821769    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmgd\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.822025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.821871    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.822025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.821899    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.822025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.822014    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.822659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.822621    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.822807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.822682    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.824611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.824589    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.845779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.845750    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmgd\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:22.887193 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:22.887163    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:23.223621 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:03:23.223585    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4fc5ea_ff9a_4c81_a2ba_bc3198dabeb6.slice/crio-beb36efd1a75790be12fbdd6edd235edc81d4e8a511b52c28f80c1dc37bd2cc7 WatchSource:0}: Error finding container beb36efd1a75790be12fbdd6edd235edc81d4e8a511b52c28f80c1dc37bd2cc7: Status 404 returned error can't find the container with id beb36efd1a75790be12fbdd6edd235edc81d4e8a511b52c28f80c1dc37bd2cc7
Nov 02 08:03:23.225113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.225088    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:03:23.529369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.529335    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:03:23.533732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.533710    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.552874 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.552637    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-cce518e50d6f\""
Nov 02 08:03:23.555155 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.555129    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:03:23.561943 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.561919    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:03:23.562375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.562302    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:03:23.562375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.562340    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:03:23.562563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.562444    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:03:23.562563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.562449    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:03:23.562801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.562777    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:03:23.633903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.633862    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjnb\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.634096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.634033    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.634203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.634183    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.634246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.634213    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.735872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.735831    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.736059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.735955    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.736059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.735977    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.736149 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.736076    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjnb\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.736692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.736668    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.736815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.736720    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.738564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.738542    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.760113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.760088    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjnb\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.848625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.848531    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:23.929280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.927999    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" event={"ID":"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6","Type":"ContainerStarted","Data":"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"}
Nov 02 08:03:23.929280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.928065    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" event={"ID":"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6","Type":"ContainerStarted","Data":"beb36efd1a75790be12fbdd6edd235edc81d4e8a511b52c28f80c1dc37bd2cc7"}
Nov 02 08:03:23.929280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.929211    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:23.971478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:23.971417    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" podStartSLOduration=1.971397739 podStartE2EDuration="1.971397739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:23.970362794 +0000 UTC m=+2348.732799700" watchObservedRunningTime="2026-11-02 08:03:23.971397739 +0000 UTC m=+2348.733834645"
Nov 02 08:03:24.154584 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:03:24.154550    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6166ba13_170c_4d6a_bd88_a1d100f3d281.slice/crio-e6b5f03c14ddb7aadfe9688a6511f4af8b3881697624366a9b31884280fd555c WatchSource:0}: Error finding container e6b5f03c14ddb7aadfe9688a6511f4af8b3881697624366a9b31884280fd555c: Status 404 returned error can't find the container with id e6b5f03c14ddb7aadfe9688a6511f4af8b3881697624366a9b31884280fd555c
Nov 02 08:03:24.156188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:24.156161    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:03:24.970917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:24.970691    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" event={"ID":"6166ba13-170c-4d6a-bd88-a1d100f3d281","Type":"ContainerStarted","Data":"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"}
Nov 02 08:03:24.970917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:24.970763    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" event={"ID":"6166ba13-170c-4d6a-bd88-a1d100f3d281","Type":"ContainerStarted","Data":"e6b5f03c14ddb7aadfe9688a6511f4af8b3881697624366a9b31884280fd555c"}
Nov 02 08:03:24.971514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:24.971005    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:25.017931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.017870    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" podStartSLOduration=2.017855392 podStartE2EDuration="2.017855392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:25.013114825 +0000 UTC m=+2349.775551752" watchObservedRunningTime="2026-11-02 08:03:25.017855392 +0000 UTC m=+2349.780292324"
Nov 02 08:03:25.232889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.232818    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:03:25.362083 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362050    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362126    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362196    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362212    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362232    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362548 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362421    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362548 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362477    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362548 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362510    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362590    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\") pod \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\" (UID: \"ace90e6c-6b77-4cd6-9a5d-d4355434a67e\") "
Nov 02 08:03:25.362718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362619    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.362718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362641    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.362930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362879    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.363015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.362988    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:25.363074 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363023    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.363354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363332    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.363411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363360    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.363411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363379    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.363411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363393    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.363513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.363408    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.364620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.364597    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.364856 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.364836    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:03:25.364993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.364968    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "kube-api-access-m4lmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:25.365063 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.364987    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token" pod "ace90e6c-6b77-4cd6-9a5d-d4355434a67e" (UID: "ace90e6c-6b77-4cd6-9a5d-d4355434a67e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:25.465446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.465402    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.465446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.465433    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4lmk\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-kube-api-access-m4lmk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.465446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.465448    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.465683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.465459    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace90e6c-6b77-4cd6-9a5d-d4355434a67e-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.979084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.979048    4515 generic.go:356] "Generic (PLEG): container finished" podID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerID="ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb" exitCode=0
Nov 02 08:03:25.979635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.979110    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"
Nov 02 08:03:25.979635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.979128    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerDied","Data":"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"}
Nov 02 08:03:25.979635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.979179    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6" event={"ID":"ace90e6c-6b77-4cd6-9a5d-d4355434a67e","Type":"ContainerDied","Data":"e5e8617b36fd16ba714f63c92c3ff1f6442f65c9b3b9193e5917693fe0da28c4"}
Nov 02 08:03:25.979635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.979204    4515 scope.go:122] "RemoveContainer" containerID="ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"
Nov 02 08:03:25.990640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:25.990613    4515 scope.go:122] "RemoveContainer" containerID="1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"
Nov 02 08:03:26.000765 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.000748    4515 scope.go:122] "RemoveContainer" containerID="ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"
Nov 02 08:03:26.001112 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:03:26.001085    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb\": container with ID starting with ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb not found: ID does not exist" containerID="ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"
Nov 02 08:03:26.001194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.001122    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb"} err="failed to get container status \"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb\": rpc error: code = NotFound desc = could not find container \"ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb\": container with ID starting with ef30d76739c750c07dd04e4bbd1c6b0d950cd28f7016368ac48ffa8e4f298bdb not found: ID does not exist"
Nov 02 08:03:26.001194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.001150    4515 scope.go:122] "RemoveContainer" containerID="1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"
Nov 02 08:03:26.001528 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:03:26.001508    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d\": container with ID starting with 1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d not found: ID does not exist" containerID="1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"
Nov 02 08:03:26.001613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.001537    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d"} err="failed to get container status \"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d\": rpc error: code = NotFound desc = could not find container \"1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d\": container with ID starting with 1285b1e0dc30db2bc20e9080ba3ae1f655a911d7aeb6c8c74a9091c3edf8d86d not found: ID does not exist"
Nov 02 08:03:26.046723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.046685    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:03:26.059162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:26.059129    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-q5fc6"]
Nov 02 08:03:27.375524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:27.375487    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" path="/var/lib/kubelet/pods/ace90e6c-6b77-4cd6-9a5d-d4355434a67e/volumes"
Nov 02 08:03:35.982598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:35.982567    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:03:35.983077 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:35.982651    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:03:36.059053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:36.058993    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:03:40.177986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.177950    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:03:40.178748 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178730    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.178797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178749    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.178797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178786    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.178797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178790    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.178910 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178898    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.178957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.178916    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace90e6c-6b77-4cd6-9a5d-d4355434a67e" containerName="istio-proxy"
Nov 02 08:03:40.184172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.184150    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.201993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.201964    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-643355-openshift-default-dockercfg-7ch4b\""
Nov 02 08:03:40.218292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.218258    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:03:40.297048 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297007    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297072    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297213    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297249    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297482 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297460    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297506    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297549    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.297652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.297639    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4956\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398450    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398561    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398591    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398674    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398849    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398912    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398942    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.398986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.398984    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399020    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399074    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399094    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4956\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399150    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399301    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.399812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.399788    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.401233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.401211    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.401608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.401584    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.424327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.424285    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.424968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.424942    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4956\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.498682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.498657    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:40.856765 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:03:40.856729    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b3f117_dd01_41d7_bf5f_66b45b3af982.slice/crio-852deccd11d58c1e6ccf0d45916013f0bb76ef70e610a3c8edc0a21396e40049 WatchSource:0}: Error finding container 852deccd11d58c1e6ccf0d45916013f0bb76ef70e610a3c8edc0a21396e40049: Status 404 returned error can't find the container with id 852deccd11d58c1e6ccf0d45916013f0bb76ef70e610a3c8edc0a21396e40049
Nov 02 08:03:40.857652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.857624    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:03:40.859928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.859895    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:40.859997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.859970    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:40.860052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:40.860026    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:41.059795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.059744    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" podUID="5a81c065-9080-4fa6-89f7-06b0a14749e8" containerName="payload-processing" containerID="cri-o://95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973" gracePeriod=25
Nov 02 08:03:41.089914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.089877    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" event={"ID":"25b3f117-dd01-41d7-bf5f-66b45b3af982","Type":"ContainerStarted","Data":"85764a84f101f826af9d202c8eeaaa50bda85759b0ee00a8ac6397c7d814a887"}
Nov 02 08:03:41.090076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.089945    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" event={"ID":"25b3f117-dd01-41d7-bf5f-66b45b3af982","Type":"ContainerStarted","Data":"852deccd11d58c1e6ccf0d45916013f0bb76ef70e610a3c8edc0a21396e40049"}
Nov 02 08:03:41.137390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.137276    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podStartSLOduration=1.137260758 podStartE2EDuration="1.137260758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:41.136555748 +0000 UTC m=+2365.898992657" watchObservedRunningTime="2026-11-02 08:03:41.137260758 +0000 UTC m=+2365.899697664"
Nov 02 08:03:41.395098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.395032    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:03:41.499257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.499217    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:41.500574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.500545    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:41.500710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.500596    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:41.518989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.518954    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\") pod \"5a81c065-9080-4fa6-89f7-06b0a14749e8\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") "
Nov 02 08:03:41.519209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.519192    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\") pod \"5a81c065-9080-4fa6-89f7-06b0a14749e8\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") "
Nov 02 08:03:41.519280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.519251    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\") pod \"5a81c065-9080-4fa6-89f7-06b0a14749e8\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") "
Nov 02 08:03:41.519380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.519285    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\") pod \"5a81c065-9080-4fa6-89f7-06b0a14749e8\" (UID: \"5a81c065-9080-4fa6-89f7-06b0a14749e8\") "
Nov 02 08:03:41.519433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.519348    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume" pod "5a81c065-9080-4fa6-89f7-06b0a14749e8" (UID: "5a81c065-9080-4fa6-89f7-06b0a14749e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:41.519677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.519653    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume" pod "5a81c065-9080-4fa6-89f7-06b0a14749e8" (UID: "5a81c065-9080-4fa6-89f7-06b0a14749e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:41.520043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.520020    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.520129 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.520047    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a81c065-9080-4fa6-89f7-06b0a14749e8-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.521692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.521666    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials" pod "5a81c065-9080-4fa6-89f7-06b0a14749e8" (UID: "5a81c065-9080-4fa6-89f7-06b0a14749e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:41.521802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.521788    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6" pod "5a81c065-9080-4fa6-89f7-06b0a14749e8" (UID: "5a81c065-9080-4fa6-89f7-06b0a14749e8"). InnerVolumeSpecName "kube-api-access-pnpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:41.642719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.642665    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.642719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:41.642718    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnpv6\" (UniqueName: \"kubernetes.io/projected/5a81c065-9080-4fa6-89f7-06b0a14749e8-kube-api-access-pnpv6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:03:42.100658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.100619    4515 generic.go:356] "Generic (PLEG): container finished" podID="5a81c065-9080-4fa6-89f7-06b0a14749e8" containerID="95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973" exitCode=0
Nov 02 08:03:42.100848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.100670    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"
Nov 02 08:03:42.100848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.100702    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" event={"ID":"5a81c065-9080-4fa6-89f7-06b0a14749e8","Type":"ContainerDied","Data":"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"}
Nov 02 08:03:42.100848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.100759    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t" event={"ID":"5a81c065-9080-4fa6-89f7-06b0a14749e8","Type":"ContainerDied","Data":"cd16d27173935604fdc0680208bdba8382fba79bddaf8d113488a8742000b3bb"}
Nov 02 08:03:42.100848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.100789    4515 scope.go:122] "RemoveContainer" containerID="95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"
Nov 02 08:03:42.114412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.114397    4515 scope.go:122] "RemoveContainer" containerID="95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"
Nov 02 08:03:42.114772 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:03:42.114749    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973\": container with ID starting with 95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973 not found: ID does not exist" containerID="95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"
Nov 02 08:03:42.114839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.114785    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973"} err="failed to get container status \"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973\": rpc error: code = NotFound desc = could not find container \"95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973\": container with ID starting with 95b21eed5304513cc25ad270b6ebb21e92813f90c8b47622eb380c84d77ce973 not found: ID does not exist"
Nov 02 08:03:42.162194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.162156    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:03:42.165725 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.165703    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89rwx2t"]
Nov 02 08:03:42.499623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.499598    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:42.500073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:42.499650    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:43.375675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:43.375641    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a81c065-9080-4fa6-89f7-06b0a14749e8" path="/var/lib/kubelet/pods/5a81c065-9080-4fa6-89f7-06b0a14749e8/volumes"
Nov 02 08:03:43.503332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:43.499740    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:43.503332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:43.499820    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:44.504491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:44.501732    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:44.504491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:44.501801    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:45.502648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:45.502601    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:45.502962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:45.502920    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:46.499688 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:46.499646    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 02 08:03:46.500188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:46.499705    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 02 08:03:47.754617 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:03:47.754576    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:03:47.964169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:47.964127    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:47.964652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:47.964630    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:47.965727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:47.965699    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:03:54.413486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:54.413444    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:03:54.413921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:54.413520    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:03:55.302364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.302324    4515 generic.go:356] "Generic (PLEG): container finished" podID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerID="95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4" exitCode=0
Nov 02 08:03:55.302364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.302342    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerDied","Data":"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"}
Nov 02 08:03:55.302841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.302823    4515 scope.go:122] "RemoveContainer" containerID="95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"
Nov 02 08:03:55.304866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.304830    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:55.304985 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.304935    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:55.305052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:55.304986    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:56.311107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:56.311072    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerStarted","Data":"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"}
Nov 02 08:03:56.435919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:56.435883    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:03:56.436107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:56.436083    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:03:56.436171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:56.436138    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:03:57.441739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:57.441694    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:03:57.442292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:57.441754    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:03:58.440062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:58.439625    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:03:58.440062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:58.439685    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:03:59.436735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:59.436696    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:03:59.437373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:03:59.437341    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:04:00.211654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.200889    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:04:00.211654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.202278    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5a81c065-9080-4fa6-89f7-06b0a14749e8" containerName="payload-processing"
Nov 02 08:04:00.211654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.202296    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a81c065-9080-4fa6-89f7-06b0a14749e8" containerName="payload-processing"
Nov 02 08:04:00.211654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.203211    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a81c065-9080-4fa6-89f7-06b0a14749e8" containerName="payload-processing"
Nov 02 08:04:00.211654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.208400    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.254635    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255347    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-71a41c\""
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255559    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"kube-root-ca.crt\""
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255632    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"agc-cred-2e46da42705f\""
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255723    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-e2e--99e06747-dockercfg-ssmlv\""
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255801    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"openshift-service-ca.crt\""
Nov 02 08:04:00.257333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.255872    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"routing-overlay\""
Nov 02 08:04:00.269349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.267852    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.269349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.267923    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.269349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.268026    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qvn\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.269349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.268207    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.369538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.369500    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.369733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.369603    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.369733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.369639    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.369733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.369727    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qvn\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.370399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.370369    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.370507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.370450    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.372361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.372338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.438577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.429115    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qvn\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.439160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.436556    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:04:00.447401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.447359    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:04:00.567617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.567578    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:00.944868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:00.944835    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:04:00.945121 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:04:00.945095    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aedee51_35c5_423c_b153_52215c44d540.slice/crio-2137d73696d1696f69d052c421e91b4afc758b7699548343e7104e0c22501bbd WatchSource:0}: Error finding container 2137d73696d1696f69d052c421e91b4afc758b7699548343e7104e0c22501bbd: Status 404 returned error can't find the container with id 2137d73696d1696f69d052c421e91b4afc758b7699548343e7104e0c22501bbd
Nov 02 08:04:01.375675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.375637    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" event={"ID":"8aedee51-35c5-423c-b153-52215c44d540","Type":"ContainerStarted","Data":"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"}
Nov 02 08:04:01.375876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.375702    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:01.375876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.375719    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" event={"ID":"8aedee51-35c5-423c-b153-52215c44d540","Type":"ContainerStarted","Data":"2137d73696d1696f69d052c421e91b4afc758b7699548343e7104e0c22501bbd"}
Nov 02 08:04:01.414273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.414215    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" podStartSLOduration=1.414200543 podStartE2EDuration="1.414200543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:04:01.412089855 +0000 UTC m=+2386.174526760" watchObservedRunningTime="2026-11-02 08:04:01.414200543 +0000 UTC m=+2386.176637448"
Nov 02 08:04:01.441336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.440739    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 08:04:01.441336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:01.440822    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 08:04:02.261495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:02.261453    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:04:02.517059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:02.516990    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:04:02.517524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:02.517506    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:04:02.518529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:02.518503    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:04:03.541560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:03.541526    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:04:04.561007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:04.560970    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:04:08.542126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.542080    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" podUID="6166ba13-170c-4d6a-bd88-a1d100f3d281" containerName="payload-processing" containerID="cri-o://5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d" gracePeriod=25
Nov 02 08:04:08.820287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.820262    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:04:08.906450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906420    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\") pod \"6166ba13-170c-4d6a-bd88-a1d100f3d281\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") "
Nov 02 08:04:08.906657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906491    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\") pod \"6166ba13-170c-4d6a-bd88-a1d100f3d281\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") "
Nov 02 08:04:08.906657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906541    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\") pod \"6166ba13-170c-4d6a-bd88-a1d100f3d281\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") "
Nov 02 08:04:08.906657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906562    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\") pod \"6166ba13-170c-4d6a-bd88-a1d100f3d281\" (UID: \"6166ba13-170c-4d6a-bd88-a1d100f3d281\") "
Nov 02 08:04:08.906852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906836    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume" pod "6166ba13-170c-4d6a-bd88-a1d100f3d281" (UID: "6166ba13-170c-4d6a-bd88-a1d100f3d281"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:04:08.906961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.906934    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume" pod "6166ba13-170c-4d6a-bd88-a1d100f3d281" (UID: "6166ba13-170c-4d6a-bd88-a1d100f3d281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:04:08.907196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.907182    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:08.907269 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.907197    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6166ba13-170c-4d6a-bd88-a1d100f3d281-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:08.908900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.908874    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials" pod "6166ba13-170c-4d6a-bd88-a1d100f3d281" (UID: "6166ba13-170c-4d6a-bd88-a1d100f3d281"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:04:08.909025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:08.908928    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb" pod "6166ba13-170c-4d6a-bd88-a1d100f3d281" (UID: "6166ba13-170c-4d6a-bd88-a1d100f3d281"). InnerVolumeSpecName "kube-api-access-nmjnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:04:09.008966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.008928    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjnb\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-kube-api-access-nmjnb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:09.008966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.008958    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6166ba13-170c-4d6a-bd88-a1d100f3d281-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:09.425377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.425343    4515 generic.go:356] "Generic (PLEG): container finished" podID="6166ba13-170c-4d6a-bd88-a1d100f3d281" containerID="5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d" exitCode=0
Nov 02 08:04:09.425594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.425401    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"
Nov 02 08:04:09.425594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.425436    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" event={"ID":"6166ba13-170c-4d6a-bd88-a1d100f3d281","Type":"ContainerDied","Data":"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"}
Nov 02 08:04:09.425594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.425507    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc" event={"ID":"6166ba13-170c-4d6a-bd88-a1d100f3d281","Type":"ContainerDied","Data":"e6b5f03c14ddb7aadfe9688a6511f4af8b3881697624366a9b31884280fd555c"}
Nov 02 08:04:09.425594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.425536    4515 scope.go:122] "RemoveContainer" containerID="5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"
Nov 02 08:04:09.438225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.438203    4515 scope.go:122] "RemoveContainer" containerID="5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"
Nov 02 08:04:09.438590 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:04:09.438559    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d\": container with ID starting with 5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d not found: ID does not exist" containerID="5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"
Nov 02 08:04:09.438681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.438601    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d"} err="failed to get container status \"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d\": rpc error: code = NotFound desc = could not find container \"5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d\": container with ID starting with 5a49bebfece43981a73bf5abb1efeaceb633c301e8de62351164df7795363a9d not found: ID does not exist"
Nov 02 08:04:09.491088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.491056    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:04:09.497778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.497749    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8b7fhc"]
Nov 02 08:04:09.562144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.562066    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" podUID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" containerName="payload-processing" containerID="cri-o://26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d" gracePeriod=25
Nov 02 08:04:09.889625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:09.889599    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:04:10.021537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.021503    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\") pod \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") "
Nov 02 08:04:10.021700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.021676    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\") pod \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") "
Nov 02 08:04:10.021792 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.021780    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\") pod \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") "
Nov 02 08:04:10.021851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.021840    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\") pod \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\" (UID: \"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6\") "
Nov 02 08:04:10.022174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.022145    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume" pod "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" (UID: "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:04:10.022405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.022369    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:10.022405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.022369    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume" pod "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" (UID: "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:04:10.023829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.023802    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd" pod "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" (UID: "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6"). InnerVolumeSpecName "kube-api-access-ptmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:04:10.023935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.023849    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials" pod "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" (UID: "ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:04:10.123836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.123793    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:10.123836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.123833    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmgd\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-kube-api-access-ptmgd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:10.123836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.123844    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:04:10.435031 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.434942    4515 generic.go:356] "Generic (PLEG): container finished" podID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" containerID="26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d" exitCode=0
Nov 02 08:04:10.435031 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.434989    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"
Nov 02 08:04:10.435237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.435025    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" event={"ID":"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6","Type":"ContainerDied","Data":"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"}
Nov 02 08:04:10.435237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.435076    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62" event={"ID":"ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6","Type":"ContainerDied","Data":"beb36efd1a75790be12fbdd6edd235edc81d4e8a511b52c28f80c1dc37bd2cc7"}
Nov 02 08:04:10.435237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.435103    4515 scope.go:122] "RemoveContainer" containerID="26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"
Nov 02 08:04:10.446736 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.446720    4515 scope.go:122] "RemoveContainer" containerID="26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"
Nov 02 08:04:10.447031 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:04:10.447012    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d\": container with ID starting with 26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d not found: ID does not exist" containerID="26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"
Nov 02 08:04:10.447124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.447038    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d"} err="failed to get container status \"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d\": rpc error: code = NotFound desc = could not find container \"26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d\": container with ID starting with 26ff33c66f9aaadc6f095134dd1916d4d1a97dd1fe848b44a6682ec8dfc4899d not found: ID does not exist"
Nov 02 08:04:10.489618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.489585    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:04:10.502654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:10.502618    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-68964wf62"]
Nov 02 08:04:11.374346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:11.374298    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6166ba13-170c-4d6a-bd88-a1d100f3d281" path="/var/lib/kubelet/pods/6166ba13-170c-4d6a-bd88-a1d100f3d281/volumes"
Nov 02 08:04:11.374815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:11.374801    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" path="/var/lib/kubelet/pods/ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6/volumes"
Nov 02 08:04:12.380036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:12.380006    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:04:17.723814 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:04:17.723770    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:04:47.723686 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:04:47.723647    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:04:53.077929 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:53.077891    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:04:55.026162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.026125    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:04:55.027584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027554    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" containerName="payload-processing"
Nov 02 08:04:55.027584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027583    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" containerName="payload-processing"
Nov 02 08:04:55.027819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027605    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6166ba13-170c-4d6a-bd88-a1d100f3d281" containerName="payload-processing"
Nov 02 08:04:55.027819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027613    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6166ba13-170c-4d6a-bd88-a1d100f3d281" containerName="payload-processing"
Nov 02 08:04:55.027939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027852    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6166ba13-170c-4d6a-bd88-a1d100f3d281" containerName="payload-processing"
Nov 02 08:04:55.027939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.027887    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4fc5ea-ff9a-4c81-a2ba-bc3198dabeb6" containerName="payload-processing"
Nov 02 08:04:55.032573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.032551    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.049200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.049165    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2365fa-openshift-default-dockercfg-b77fw\""
Nov 02 08:04:55.049606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.049584    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.049883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.049663    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.049883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.049815    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.049978    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.050009    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.050099    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.050136    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050359 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.050223    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqf9\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.050431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.050369    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.059194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.059168    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:04:55.152437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.152324    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.152819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.152713    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.153329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.153260    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.153556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.153537    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.153758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.153743    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.153871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.153856    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.153993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.153979    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.154174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.154159    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.154353 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.154335    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqf9\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.156567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.156503    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.156719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.156694    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.157135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.157113    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.157429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.157403    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.157688 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.157666    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.160729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.160686    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.217710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.216745    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.273333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.260046    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqf9\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.294374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.292671    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\") pod \"e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") " pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.355335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.351201    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:55.699098 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:04:55.699058    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b49d3d4_fa91_4336_a255_d691720f477e.slice/crio-61430da52b4e4fd1cd51f00efc6b86f3d4186623d91f2bf8751757cfc6f3a744 WatchSource:0}: Error finding container 61430da52b4e4fd1cd51f00efc6b86f3d4186623d91f2bf8751757cfc6f3a744: Status 404 returned error can't find the container with id 61430da52b4e4fd1cd51f00efc6b86f3d4186623d91f2bf8751757cfc6f3a744
Nov 02 08:04:55.699319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.699285    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:04:55.701886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.701851    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:55.701994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.701952    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:55.702084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.701997    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:55.864414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.864378    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerStarted","Data":"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"}
Nov 02 08:04:55.864605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.864436    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerStarted","Data":"61430da52b4e4fd1cd51f00efc6b86f3d4186623d91f2bf8751757cfc6f3a744"}
Nov 02 08:04:55.913587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:55.913524    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podStartSLOduration=1.91350449 podStartE2EDuration="1.91350449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:04:55.910736933 +0000 UTC m=+2440.673173839" watchObservedRunningTime="2026-11-02 08:04:55.91350449 +0000 UTC m=+2440.675941396"
Nov 02 08:04:56.351854 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:56.351823    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:56.358735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:56.358706    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:56.871394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:56.871359    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:04:56.872515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:04:56.872498    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:05:00.139685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.139647    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:05:00.144410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.144382    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.172604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172571    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-0687728475e9\""
Nov 02 08:05:00.172790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172685    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:05:00.172790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172745    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:05:00.172936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172820    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:05:00.172936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172878    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:05:00.173065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.172980    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-74b36e68d5bd\""
Nov 02 08:05:00.173123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.173079    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-4nccn\""
Nov 02 08:05:00.178217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.178185    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:05:00.249610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.249570    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.249819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.249740    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.250006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.249977    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7km7\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.250082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.250025    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.351282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.351245    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7km7\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.351282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.351286    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.351575 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.351372    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.351575 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.351467    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.352181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.352151    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.352431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.352332    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.354267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.354241    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.376026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.375989    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7km7\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.460578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.460488    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.775469 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:05:00.775437    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd485ec8c_7df5_47ee_ba10_b112ec94a1af.slice/crio-259a0eb2c5a876e54af4e539c7f559b467c00ff27272e21075de9d1d6b9a78da WatchSource:0}: Error finding container 259a0eb2c5a876e54af4e539c7f559b467c00ff27272e21075de9d1d6b9a78da: Status 404 returned error can't find the container with id 259a0eb2c5a876e54af4e539c7f559b467c00ff27272e21075de9d1d6b9a78da
Nov 02 08:05:00.776131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.776107    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:05:00.987805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.987765    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" event={"ID":"d485ec8c-7df5-47ee-ba10-b112ec94a1af","Type":"ContainerStarted","Data":"eae53eb663a1a63e5763b1a6ee04df3f58aefe2f781762f4674721fbcf6720a2"}
Nov 02 08:05:00.987993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.987837    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:00.987993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:00.987862    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" event={"ID":"d485ec8c-7df5-47ee-ba10-b112ec94a1af","Type":"ContainerStarted","Data":"259a0eb2c5a876e54af4e539c7f559b467c00ff27272e21075de9d1d6b9a78da"}
Nov 02 08:05:01.034029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:01.033919    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" podStartSLOduration=1.033902474 podStartE2EDuration="1.033902474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:01.029521319 +0000 UTC m=+2445.791958224" watchObservedRunningTime="2026-11-02 08:05:01.033902474 +0000 UTC m=+2445.796339420"
Nov 02 08:05:02.978210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:02.978169    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:02.983191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:02.983167    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012186    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012239    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012293    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012339    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012505    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-cce518e50d6f\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.012540    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:05:03.016676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.013491    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:05:03.018681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.018650    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:03.038054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.037537    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.038054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.037643    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.038054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.037674    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.038054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.037727    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2b5\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.139600 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.139514    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2b5\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.139846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.139827    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.139937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.139919    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.139994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.139943    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.140597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.140561    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.140805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.140785    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.142531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.142512    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.184728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.184701    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2b5\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\") pod \"payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.300864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.300825    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:03.638582 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:05:03.638542    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97debb_5d43_498d_970b_35c3aec4a01a.slice/crio-ae4e3ff744d020933891e14ae591bb85b3c07a9305dbc6eb3a04d266a686ebe3 WatchSource:0}: Error finding container ae4e3ff744d020933891e14ae591bb85b3c07a9305dbc6eb3a04d266a686ebe3: Status 404 returned error can't find the container with id ae4e3ff744d020933891e14ae591bb85b3c07a9305dbc6eb3a04d266a686ebe3
Nov 02 08:05:03.640868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:03.640839    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:04.012113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:04.012075    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" event={"ID":"3d97debb-5d43-498d-970b-35c3aec4a01a","Type":"ContainerStarted","Data":"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"}
Nov 02 08:05:04.012625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:04.012131    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" event={"ID":"3d97debb-5d43-498d-970b-35c3aec4a01a","Type":"ContainerStarted","Data":"ae4e3ff744d020933891e14ae591bb85b3c07a9305dbc6eb3a04d266a686ebe3"}
Nov 02 08:05:04.012625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:04.012224    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:04.059373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:04.059301    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" podStartSLOduration=2.059285871 podStartE2EDuration="2.059285871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:04.054671099 +0000 UTC m=+2448.817108004" watchObservedRunningTime="2026-11-02 08:05:04.059285871 +0000 UTC m=+2448.821722804"
Nov 02 08:05:11.998666 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:11.998630    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:05:15.021527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:15.021497    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:17.768599 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:05:17.768555    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:05:38.924673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:38.924589    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:05:38.976995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:38.976959    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:43.925635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:43.925591    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" podUID="8aedee51-35c5-423c-b153-52215c44d540" containerName="payload-processing" containerID="cri-o://6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664" gracePeriod=25
Nov 02 08:05:43.980650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:43.980609    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" podUID="3d97debb-5d43-498d-970b-35c3aec4a01a" containerName="payload-processing" containerID="cri-o://e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c" gracePeriod=25
Nov 02 08:05:44.284652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.284621    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:05:44.310944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.310920    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:44.384118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384076    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\") pod \"3d97debb-5d43-498d-970b-35c3aec4a01a\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") "
Nov 02 08:05:44.384374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384168    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\") pod \"8aedee51-35c5-423c-b153-52215c44d540\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") "
Nov 02 08:05:44.384374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384208    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\") pod \"8aedee51-35c5-423c-b153-52215c44d540\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") "
Nov 02 08:05:44.384374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384239    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\") pod \"3d97debb-5d43-498d-970b-35c3aec4a01a\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") "
Nov 02 08:05:44.384374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384349    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\") pod \"3d97debb-5d43-498d-970b-35c3aec4a01a\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") "
Nov 02 08:05:44.384668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384418    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\") pod \"8aedee51-35c5-423c-b153-52215c44d540\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") "
Nov 02 08:05:44.384668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384451    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\") pod \"3d97debb-5d43-498d-970b-35c3aec4a01a\" (UID: \"3d97debb-5d43-498d-970b-35c3aec4a01a\") "
Nov 02 08:05:44.384796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.384691    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\") pod \"8aedee51-35c5-423c-b153-52215c44d540\" (UID: \"8aedee51-35c5-423c-b153-52215c44d540\") "
Nov 02 08:05:44.387460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.387249    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume" pod "8aedee51-35c5-423c-b153-52215c44d540" (UID: "8aedee51-35c5-423c-b153-52215c44d540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:44.393126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.387878    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume" pod "3d97debb-5d43-498d-970b-35c3aec4a01a" (UID: "3d97debb-5d43-498d-970b-35c3aec4a01a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:44.393126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.388004    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume" pod "8aedee51-35c5-423c-b153-52215c44d540" (UID: "8aedee51-35c5-423c-b153-52215c44d540"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:44.393126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.388381    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume" pod "3d97debb-5d43-498d-970b-35c3aec4a01a" (UID: "3d97debb-5d43-498d-970b-35c3aec4a01a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:44.393126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.390477    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials" pod "8aedee51-35c5-423c-b153-52215c44d540" (UID: "8aedee51-35c5-423c-b153-52215c44d540"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393763    4515 generic.go:356] "Generic (PLEG): container finished" podID="3d97debb-5d43-498d-970b-35c3aec4a01a" containerID="e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c" exitCode=0
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393855    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn" pod "8aedee51-35c5-423c-b153-52215c44d540" (UID: "8aedee51-35c5-423c-b153-52215c44d540"). InnerVolumeSpecName "kube-api-access-n2qvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393907    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393904    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" event={"ID":"3d97debb-5d43-498d-970b-35c3aec4a01a","Type":"ContainerDied","Data":"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"}
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393964    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5" pod "3d97debb-5d43-498d-970b-35c3aec4a01a" (UID: "3d97debb-5d43-498d-970b-35c3aec4a01a"). InnerVolumeSpecName "kube-api-access-xc2b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.393973    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8" event={"ID":"3d97debb-5d43-498d-970b-35c3aec4a01a","Type":"ContainerDied","Data":"ae4e3ff744d020933891e14ae591bb85b3c07a9305dbc6eb3a04d266a686ebe3"}
Nov 02 08:05:44.395539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.394005    4515 scope.go:122] "RemoveContainer" containerID="e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"
Nov 02 08:05:44.396230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.396177    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials" pod "3d97debb-5d43-498d-970b-35c3aec4a01a" (UID: "3d97debb-5d43-498d-970b-35c3aec4a01a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:44.396748 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.396717    4515 generic.go:356] "Generic (PLEG): container finished" podID="8aedee51-35c5-423c-b153-52215c44d540" containerID="6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664" exitCode=0
Nov 02 08:05:44.396856 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.396820    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" event={"ID":"8aedee51-35c5-423c-b153-52215c44d540","Type":"ContainerDied","Data":"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"}
Nov 02 08:05:44.396921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.396871    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq" event={"ID":"8aedee51-35c5-423c-b153-52215c44d540","Type":"ContainerDied","Data":"2137d73696d1696f69d052c421e91b4afc758b7699548343e7104e0c22501bbd"}
Nov 02 08:05:44.396977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.396943    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"
Nov 02 08:05:44.411297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.411240    4515 scope.go:122] "RemoveContainer" containerID="e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"
Nov 02 08:05:44.412182 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:05:44.411809    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c\": container with ID starting with e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c not found: ID does not exist" containerID="e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"
Nov 02 08:05:44.412182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.411862    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c"} err="failed to get container status \"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c\": rpc error: code = NotFound desc = could not find container \"e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c\": container with ID starting with e3de77e1eeae29dcbffac8003249af4740f6dec14760126f786bc03f45fc784c not found: ID does not exist"
Nov 02 08:05:44.412182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.411907    4515 scope.go:122] "RemoveContainer" containerID="6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"
Nov 02 08:05:44.424755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.424735    4515 scope.go:122] "RemoveContainer" containerID="6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"
Nov 02 08:05:44.425103 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:05:44.425079    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664\": container with ID starting with 6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664 not found: ID does not exist" containerID="6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"
Nov 02 08:05:44.425194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.425113    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664"} err="failed to get container status \"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664\": rpc error: code = NotFound desc = could not find container \"6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664\": container with ID starting with 6559d9e32cef9fccf1b7381744d18e443a3bd0d431195a957b0f032ce3bfa664 not found: ID does not exist"
Nov 02 08:05:44.464776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.464596    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:05:44.470801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.470772    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656drqndq"]
Nov 02 08:05:44.486956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.486917    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.486956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.486956    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.486970    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qvn\" (UniqueName: \"kubernetes.io/projected/8aedee51-35c5-423c-b153-52215c44d540-kube-api-access-n2qvn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.486984    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.486998    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc2b5\" (UniqueName: \"kubernetes.io/projected/3d97debb-5d43-498d-970b-35c3aec4a01a-kube-api-access-xc2b5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.487010    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.487024    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aedee51-35c5-423c-b153-52215c44d540-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.487205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.487037    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d97debb-5d43-498d-970b-35c3aec4a01a-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.753727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.753690    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:44.759059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:44.759031    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-57f8psrz8"]
Nov 02 08:05:45.375163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:45.375128    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d97debb-5d43-498d-970b-35c3aec4a01a" path="/var/lib/kubelet/pods/3d97debb-5d43-498d-970b-35c3aec4a01a/volumes"
Nov 02 08:05:45.375659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:45.375644    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aedee51-35c5-423c-b153-52215c44d540" path="/var/lib/kubelet/pods/8aedee51-35c5-423c-b153-52215c44d540/volumes"
Nov 02 08:05:47.694380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:47.694345    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:05:47.723790 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:05:47.723751    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:05:53.168788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.168748    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:05:53.169716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169690    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8aedee51-35c5-423c-b153-52215c44d540" containerName="payload-processing"
Nov 02 08:05:53.169716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169716    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aedee51-35c5-423c-b153-52215c44d540" containerName="payload-processing"
Nov 02 08:05:53.169900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169752    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3d97debb-5d43-498d-970b-35c3aec4a01a" containerName="payload-processing"
Nov 02 08:05:53.169900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169760    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d97debb-5d43-498d-970b-35c3aec4a01a" containerName="payload-processing"
Nov 02 08:05:53.170012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169941    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d97debb-5d43-498d-970b-35c3aec4a01a" containerName="payload-processing"
Nov 02 08:05:53.170012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.169968    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aedee51-35c5-423c-b153-52215c44d540" containerName="payload-processing"
Nov 02 08:05:53.175524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.175501    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.193859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.193829    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:05:53.201012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.200975    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:05:53.201178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.201058    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-917c10c547af\""
Nov 02 08:05:53.201178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.201147    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:05:53.201178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.201159    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:05:53.201354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.201209    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:05:53.203573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.203551    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-zk7vt\""
Nov 02 08:05:53.310343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.292629    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.310343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.292688    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nlp\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.310343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.292763    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.310343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.292831    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.412334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.401837    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.430382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.430273    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nlp\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.430715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.430690    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.432911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.432879    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.434393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.434037    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.435139 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.435117    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.456716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.456684    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.461977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.461946    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nlp\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.491848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.491757    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:53.926697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:53.922655    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:05:54.505887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:54.505851    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" event={"ID":"cb162794-ba2a-4ca0-8284-dfec97f3abd9","Type":"ContainerStarted","Data":"b0ed0920d9644b1c5d6aa4aa3b07bcf111b8dc82cdd6445c8b621d2ce6680fef"}
Nov 02 08:05:54.506288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:54.505931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" event={"ID":"cb162794-ba2a-4ca0-8284-dfec97f3abd9","Type":"ContainerStarted","Data":"e0747048133aec0180244948b36d442bd3343c4af1dd90fc484abd500586c1b5"}
Nov 02 08:05:54.506288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:54.506044    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:05:54.548424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:54.548366    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" podStartSLOduration=1.548353691 podStartE2EDuration="1.548353691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:54.547282189 +0000 UTC m=+2499.309719095" watchObservedRunningTime="2026-11-02 08:05:54.548353691 +0000 UTC m=+2499.310790596"
Nov 02 08:05:58.426109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:05:58.426073    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:06:03.426873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.426831    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" podUID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" containerName="payload-processing" containerID="cri-o://b0ed0920d9644b1c5d6aa4aa3b07bcf111b8dc82cdd6445c8b621d2ce6680fef" gracePeriod=25
Nov 02 08:06:03.596839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.596503    4515 generic.go:356] "Generic (PLEG): container finished" podID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" containerID="b0ed0920d9644b1c5d6aa4aa3b07bcf111b8dc82cdd6445c8b621d2ce6680fef" exitCode=0
Nov 02 08:06:03.596839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.596757    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" event={"ID":"cb162794-ba2a-4ca0-8284-dfec97f3abd9","Type":"ContainerDied","Data":"b0ed0920d9644b1c5d6aa4aa3b07bcf111b8dc82cdd6445c8b621d2ce6680fef"}
Nov 02 08:06:03.754551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.754512    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:06:03.851766 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.851726    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\") pod \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") "
Nov 02 08:06:03.851950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.851833    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\") pod \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") "
Nov 02 08:06:03.851950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.851900    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\") pod \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") "
Nov 02 08:06:03.852127 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.852108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\") pod \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\" (UID: \"cb162794-ba2a-4ca0-8284-dfec97f3abd9\") "
Nov 02 08:06:03.852227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.852208    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume" pod "cb162794-ba2a-4ca0-8284-dfec97f3abd9" (UID: "cb162794-ba2a-4ca0-8284-dfec97f3abd9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:03.852348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.852291    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume" pod "cb162794-ba2a-4ca0-8284-dfec97f3abd9" (UID: "cb162794-ba2a-4ca0-8284-dfec97f3abd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:03.852712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.852693    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:03.852800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.852716    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb162794-ba2a-4ca0-8284-dfec97f3abd9-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:03.854286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.854262    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp" pod "cb162794-ba2a-4ca0-8284-dfec97f3abd9" (UID: "cb162794-ba2a-4ca0-8284-dfec97f3abd9"). InnerVolumeSpecName "kube-api-access-j7nlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:03.854367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.854300    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials" pod "cb162794-ba2a-4ca0-8284-dfec97f3abd9" (UID: "cb162794-ba2a-4ca0-8284-dfec97f3abd9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:03.953937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.953902    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:03.953937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:03.953935    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nlp\" (UniqueName: \"kubernetes.io/projected/cb162794-ba2a-4ca0-8284-dfec97f3abd9-kube-api-access-j7nlp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:04.608684 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:04.608645    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt" event={"ID":"cb162794-ba2a-4ca0-8284-dfec97f3abd9","Type":"ContainerDied","Data":"e0747048133aec0180244948b36d442bd3343c4af1dd90fc484abd500586c1b5"}
Nov 02 08:06:04.609173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:04.608718    4515 scope.go:122] "RemoveContainer" containerID="b0ed0920d9644b1c5d6aa4aa3b07bcf111b8dc82cdd6445c8b621d2ce6680fef"
Nov 02 08:06:04.609173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:04.608660    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"
Nov 02 08:06:04.664564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:04.664533    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:06:04.677957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:04.677922    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7657vwcmt"]
Nov 02 08:06:05.375356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:05.375293    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" path="/var/lib/kubelet/pods/cb162794-ba2a-4ca0-8284-dfec97f3abd9/volumes"
Nov 02 08:06:17.727710 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:17.727672    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:06:20.981612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.981577    4515 generic.go:356] "Generic (PLEG): container finished" podID="2b49d3d4-fa91-4336-a255-d691720f477e" containerID="c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646" exitCode=0
Nov 02 08:06:20.982047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.981647    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerDied","Data":"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"}
Nov 02 08:06:20.982179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.982158    4515 scope.go:122] "RemoveContainer" containerID="c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"
Nov 02 08:06:20.984030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.983999    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:06:20.984094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.984073    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:06:20.984131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:20.984123    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:06:21.256893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.256860    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:06:21.257915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.257897    4515 cpu_manager.go:414] "containerMap: removing container" podUID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" containerName="payload-processing"
Nov 02 08:06:21.257971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.257917    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" containerName="payload-processing"
Nov 02 08:06:21.258102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.258091    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb162794-ba2a-4ca0-8284-dfec97f3abd9" containerName="payload-processing"
Nov 02 08:06:21.262485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.262463    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.281787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.281760    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:06:21.282115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.282093    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"routing-overlay\""
Nov 02 08:06:21.283057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.282840    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"e2e-cmp-legacy-53467f-provider-cred\""
Nov 02 08:06:21.283289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.283271    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c712b1\""
Nov 02 08:06:21.283456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.283404    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"kube-root-ca.crt\""
Nov 02 08:06:21.283456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.283357    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-e2e--50b787fd-dockercfg-9klzp\""
Nov 02 08:06:21.283580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.283285    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"openshift-service-ca.crt\""
Nov 02 08:06:21.344794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.344756    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.345061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.345043    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86chh\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.345435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.345414    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.345595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.345579    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.540379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.539937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.544330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.542477    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.544330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.542544    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86chh\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.544330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.542713    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.546775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.545088    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.546775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.546115    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.570330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.567672    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.572466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.570526    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86chh\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.588613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.585813    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:21.906997 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:06:21.906962    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98915278_0a84_4b1c_9456_170ce6456b1b.slice/crio-4fe9512104ab9b37c4dea76a41c5fee18e73d3490ed322d49a61b6b99cccd146 WatchSource:0}: Error finding container 4fe9512104ab9b37c4dea76a41c5fee18e73d3490ed322d49a61b6b99cccd146: Status 404 returned error can't find the container with id 4fe9512104ab9b37c4dea76a41c5fee18e73d3490ed322d49a61b6b99cccd146
Nov 02 08:06:21.908360 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.908327    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:06:21.991372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.991342    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerStarted","Data":"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"}
Nov 02 08:06:21.993043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.993015    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" event={"ID":"98915278-0a84-4b1c-9456-170ce6456b1b","Type":"ContainerStarted","Data":"4fe9512104ab9b37c4dea76a41c5fee18e73d3490ed322d49a61b6b99cccd146"}
Nov 02 08:06:21.993203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:21.993078    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:22.083340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:22.083271    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" podStartSLOduration=1.083255901 podStartE2EDuration="1.083255901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:22.080056258 +0000 UTC m=+2526.842493164" watchObservedRunningTime="2026-11-02 08:06:22.083255901 +0000 UTC m=+2526.845692807"
Nov 02 08:06:22.351371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:22.351331    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:06:22.352832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:22.352805    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:22.352961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:22.352873    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:23.044107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:23.044064    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" event={"ID":"98915278-0a84-4b1c-9456-170ce6456b1b","Type":"ContainerStarted","Data":"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"}
Nov 02 08:06:23.351982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:23.351880    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:23.351982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:23.351950    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:24.351677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:24.351639    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:24.352107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:24.351704    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:24.853368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:24.853319    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:06:25.360361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:25.356325    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:25.360361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:25.356387    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:26.351745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:26.351709    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:26.351946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:26.351769    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:26.875449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:26.875407    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:06:27.352407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:27.352371    4515 patch_prober.go:29] interesting pod/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 02 08:06:27.352604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:27.352443    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 02 08:06:28.357474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:28.357439    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:06:28.358540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:28.358519    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:06:28.596483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:28.596435    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:06:29.856429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:29.856383    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" podUID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" containerName="payload-processing" containerID="cri-o://eae53eb663a1a63e5763b1a6ee04df3f58aefe2f781762f4674721fbcf6720a2" gracePeriod=25
Nov 02 08:06:30.141392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.139506    4515 generic.go:356] "Generic (PLEG): container finished" podID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" containerID="eae53eb663a1a63e5763b1a6ee04df3f58aefe2f781762f4674721fbcf6720a2" exitCode=0
Nov 02 08:06:30.141392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.139744    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" event={"ID":"d485ec8c-7df5-47ee-ba10-b112ec94a1af","Type":"ContainerDied","Data":"eae53eb663a1a63e5763b1a6ee04df3f58aefe2f781762f4674721fbcf6720a2"}
Nov 02 08:06:30.368069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.368036    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:06:30.433002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.432898    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\") pod \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") "
Nov 02 08:06:30.433198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.433015    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\") pod \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") "
Nov 02 08:06:30.433198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.433160    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\") pod \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") "
Nov 02 08:06:30.433362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.433261    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\") pod \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\" (UID: \"d485ec8c-7df5-47ee-ba10-b112ec94a1af\") "
Nov 02 08:06:30.433649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.433600    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume" pod "d485ec8c-7df5-47ee-ba10-b112ec94a1af" (UID: "d485ec8c-7df5-47ee-ba10-b112ec94a1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:30.433762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.433713    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume" pod "d485ec8c-7df5-47ee-ba10-b112ec94a1af" (UID: "d485ec8c-7df5-47ee-ba10-b112ec94a1af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:30.436037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.435985    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7" pod "d485ec8c-7df5-47ee-ba10-b112ec94a1af" (UID: "d485ec8c-7df5-47ee-ba10-b112ec94a1af"). InnerVolumeSpecName "kube-api-access-b7km7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:30.436172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.436059    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials" pod "d485ec8c-7df5-47ee-ba10-b112ec94a1af" (UID: "d485ec8c-7df5-47ee-ba10-b112ec94a1af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:30.436894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.436870    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.437029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.436901    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d485ec8c-7df5-47ee-ba10-b112ec94a1af-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.538820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.538784    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.538820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:30.538813    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7km7\" (UniqueName: \"kubernetes.io/projected/d485ec8c-7df5-47ee-ba10-b112ec94a1af-kube-api-access-b7km7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:06:31.150284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.150250    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"
Nov 02 08:06:31.150284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.150270    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4" event={"ID":"d485ec8c-7df5-47ee-ba10-b112ec94a1af","Type":"ContainerDied","Data":"259a0eb2c5a876e54af4e539c7f559b467c00ff27272e21075de9d1d6b9a78da"}
Nov 02 08:06:31.150922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.150354    4515 scope.go:122] "RemoveContainer" containerID="eae53eb663a1a63e5763b1a6ee04df3f58aefe2f781762f4674721fbcf6720a2"
Nov 02 08:06:31.204280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.204250    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:06:31.217016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.216987    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5bd4hhlr4"]
Nov 02 08:06:31.374922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:31.374887    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" path="/var/lib/kubelet/pods/d485ec8c-7df5-47ee-ba10-b112ec94a1af/volumes"
Nov 02 08:06:34.054833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:34.054804    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:06:43.380614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.380573    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:06:43.381505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.381485    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" containerName="payload-processing"
Nov 02 08:06:43.381505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.381505    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" containerName="payload-processing"
Nov 02 08:06:43.381621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.381603    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d485ec8c-7df5-47ee-ba10-b112ec94a1af" containerName="payload-processing"
Nov 02 08:06:43.385615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.385587    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.388816 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.388787    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:06:43.402682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.402652    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-e68e818391a6\""
Nov 02 08:06:43.414885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.414860    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:06:43.415099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.415070    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:06:43.415099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.415085    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:06:43.415259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.415125    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-kmjj5\""
Nov 02 08:06:43.415259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.415166    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:06:43.435820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.435784    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.436016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.435900    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.436088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.436067    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmtk\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.436182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.436157    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.540289    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.540408    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.540573    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmtk\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.540618    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.543122    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.558335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.543875    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.559553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.559485    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.638966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.622020    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmtk\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:43.704058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:43.703751    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:44.010144 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:06:44.010112    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc909e69a_b40e_4dbb_b5a9_c78a5ec0c0c3.slice/crio-d25f9e0ddd882d1c0647ec9ea88019a9ec1c2507f9e073b6ef925665b8c5f083 WatchSource:0}: Error finding container d25f9e0ddd882d1c0647ec9ea88019a9ec1c2507f9e073b6ef925665b8c5f083: Status 404 returned error can't find the container with id d25f9e0ddd882d1c0647ec9ea88019a9ec1c2507f9e073b6ef925665b8c5f083
Nov 02 08:06:44.012119 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:44.012090    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:06:44.308157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:44.308071    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" event={"ID":"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3","Type":"ContainerStarted","Data":"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"}
Nov 02 08:06:44.308157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:44.308123    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" event={"ID":"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3","Type":"ContainerStarted","Data":"d25f9e0ddd882d1c0647ec9ea88019a9ec1c2507f9e073b6ef925665b8c5f083"}
Nov 02 08:06:44.308374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:44.308277    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:44.353066 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.353030    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:44.353066 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.353060    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:44.353437 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.353205    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:44.853186646 +0000 UTC m=+2549.615623552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:44.355653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:44.355611    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" podStartSLOduration=1.355596541 podStartE2EDuration="1.355596541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:44.353820606 +0000 UTC m=+2549.116257512" watchObservedRunningTime="2026-11-02 08:06:44.355596541 +0000 UTC m=+2549.118033448"
Nov 02 08:06:44.874330 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.872687    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:44.874330 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.872725    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:44.874330 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:44.872801    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:45.872781515 +0000 UTC m=+2550.635218419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:45.896113 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:45.896076    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:45.896113 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:45.896107    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:45.896673 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:45.896192    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:47.896167726 +0000 UTC m=+2552.658604624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:47.725597 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:47.725561    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:06:47.923420 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:47.923387    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:47.923420 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:47.923412    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:47.923657 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:47.923464    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:06:51.923450416 +0000 UTC m=+2556.685887316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:49.801321 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.801259    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:06:49.806603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.806583    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.838099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.838045    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:06:49.838510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.838486    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:06:49.838630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.838587    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:06:49.838953 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.838936    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:06:49.839016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.839006    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:06:49.840074 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.840049    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:06:49.842516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.842494    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:06:49.865219 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.865183    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.865464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.865248    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.865464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.865388    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.865464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.865429    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.967142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.967102    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.967142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.967147    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.967433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.967375    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.967433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.967423    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.968103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.968072    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.968358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.968328    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.969973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.969955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:49.991955 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:49.991928    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94869qmk\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:50.122665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:50.122579    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:50.436704 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:06:50.436302    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc4a1b3_4580_4b6a_a79f_3d2efc0f7029.slice/crio-39be8afcba56e736dbf9830b9823e38223c12f04a616dde5066339d1f5ce2eca WatchSource:0}: Error finding container 39be8afcba56e736dbf9830b9823e38223c12f04a616dde5066339d1f5ce2eca: Status 404 returned error can't find the container with id 39be8afcba56e736dbf9830b9823e38223c12f04a616dde5066339d1f5ce2eca
Nov 02 08:06:50.438195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:50.438168    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:06:51.427584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:51.427540    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" event={"ID":"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029","Type":"ContainerStarted","Data":"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"}
Nov 02 08:06:51.428036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:51.427615    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" event={"ID":"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029","Type":"ContainerStarted","Data":"39be8afcba56e736dbf9830b9823e38223c12f04a616dde5066339d1f5ce2eca"}
Nov 02 08:06:51.428036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:51.427762    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:06:51.479755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:51.479693    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" podStartSLOduration=2.479679881 podStartE2EDuration="2.479679881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:51.477578742 +0000 UTC m=+2556.240015647" watchObservedRunningTime="2026-11-02 08:06:51.479679881 +0000 UTC m=+2556.242116842"
Nov 02 08:06:52.021333 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:52.012767    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:52.021333 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:52.012799    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:52.021333 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:06:52.012863    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:07:00.012843231 +0000 UTC m=+2564.775280128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:06:55.318627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:55.318596    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:06:57.143934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:06:57.143899    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:07:00.111878 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:00.111793    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/agc-cred-e68e818391a6: secret "agc-cred-e68e818391a6" not found
Nov 02 08:07:00.111878 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:00.111822    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r: secret "agc-cred-e68e818391a6" not found
Nov 02 08:07:00.111878 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:00.111876    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials podName:c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3 nodeName:}" failed. No retries permitted until 2026-11-02 08:07:16.111860939 +0000 UTC m=+2580.874297852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3") : secret "agc-cred-e68e818391a6" not found
Nov 02 08:07:02.144947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.144896    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" podUID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" containerName="payload-processing" containerID="cri-o://53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63" gracePeriod=25
Nov 02 08:07:02.442809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.442775    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:07:02.448258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.448230    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:07:02.542558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.542525    4515 generic.go:356] "Generic (PLEG): container finished" podID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" containerID="53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63" exitCode=0
Nov 02 08:07:02.542845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.542573    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"
Nov 02 08:07:02.542845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.542609    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" event={"ID":"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3","Type":"ContainerDied","Data":"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"}
Nov 02 08:07:02.542845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.542663    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r" event={"ID":"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3","Type":"ContainerDied","Data":"d25f9e0ddd882d1c0647ec9ea88019a9ec1c2507f9e073b6ef925665b8c5f083"}
Nov 02 08:07:02.542845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.542686    4515 scope.go:122] "RemoveContainer" containerID="53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"
Nov 02 08:07:02.554726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.554705    4515 scope.go:122] "RemoveContainer" containerID="53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"
Nov 02 08:07:02.555062 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:02.555039    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63\": container with ID starting with 53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63 not found: ID does not exist" containerID="53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"
Nov 02 08:07:02.555177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.555070    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63"} err="failed to get container status \"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63\": rpc error: code = NotFound desc = could not find container \"53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63\": container with ID starting with 53ddc234eaf6dc7f4311d36b22c9ebf39e9bbbe24ca3d32639424252d3f08d63 not found: ID does not exist"
Nov 02 08:07:02.557515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.557497    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\") pod \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") "
Nov 02 08:07:02.557583 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.557550    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\") pod \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") "
Nov 02 08:07:02.557718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.557702    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\") pod \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") "
Nov 02 08:07:02.557919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.557896    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\") pod \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\" (UID: \"c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3\") "
Nov 02 08:07:02.558057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.557933    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume" pod "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:02.558279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.558252    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume" pod "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:02.558778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.558753    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:02.558858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.558785    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:02.560815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.560786    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials" pod "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:02.561101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.561084    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk" pod "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" (UID: "c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3"). InnerVolumeSpecName "kube-api-access-4fmtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:02.660157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.660074    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:02.660157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.660104    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fmtk\" (UniqueName: \"kubernetes.io/projected/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3-kube-api-access-4fmtk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:02.919259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.919178    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:07:02.926911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:02.926881    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-848c9v92r"]
Nov 02 08:07:03.374796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:03.374753    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" path="/var/lib/kubelet/pods/c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3/volumes"
Nov 02 08:07:17.729553 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:17.729512    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:07:22.666687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:22.666650    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:07:46.179525 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:46.179484    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:07:47.727204 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:47.727166    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:07:51.180823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.180781    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" podUID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" containerName="payload-processing" containerID="cri-o://89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3" gracePeriod=25
Nov 02 08:07:51.460317 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.460281    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:07:51.522441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.522407    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\") pod \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") "
Nov 02 08:07:51.522722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.522698    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\") pod \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") "
Nov 02 08:07:51.522864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.522769    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\") pod \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") "
Nov 02 08:07:51.523246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.523024    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume" pod "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" (UID: "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:51.524297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.524268    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:51.527099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.527071    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials" pod "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" (UID: "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:51.527581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.527554    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd" pod "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" (UID: "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029"). InnerVolumeSpecName "kube-api-access-2hdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:51.625857 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.625818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\") pod \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\" (UID: \"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029\") "
Nov 02 08:07:51.626227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.626205    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume" pod "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" (UID: "5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:51.626279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.626214    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:51.626279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.626258    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-kube-api-access-2hdvd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:51.727761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:51.727676    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:07:52.090286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.090245    4515 generic.go:356] "Generic (PLEG): container finished" podID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" containerID="89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3" exitCode=0
Nov 02 08:07:52.090513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.090290    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" event={"ID":"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029","Type":"ContainerDied","Data":"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"}
Nov 02 08:07:52.090513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.090337    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"
Nov 02 08:07:52.090513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.090407    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk" event={"ID":"5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029","Type":"ContainerDied","Data":"39be8afcba56e736dbf9830b9823e38223c12f04a616dde5066339d1f5ce2eca"}
Nov 02 08:07:52.090513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.090433    4515 scope.go:122] "RemoveContainer" containerID="89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"
Nov 02 08:07:52.105419 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.105393    4515 scope.go:122] "RemoveContainer" containerID="89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"
Nov 02 08:07:52.105715 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:07:52.105692    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3\": container with ID starting with 89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3 not found: ID does not exist" containerID="89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"
Nov 02 08:07:52.105806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.105723    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3"} err="failed to get container status \"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3\": rpc error: code = NotFound desc = could not find container \"89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3\": container with ID starting with 89a91497194936dbecf79b929fee9971d984b0f9d84f2a20f22a73597e63d6f3 not found: ID does not exist"
Nov 02 08:07:52.164446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.164406    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:07:52.182210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:52.182170    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94869qmk"]
Nov 02 08:07:53.374809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:07:53.374774    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" path="/var/lib/kubelet/pods/5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029/volumes"
Nov 02 08:08:01.111252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:01.111216    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:08:17.789492 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:08:17.789450    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:08:21.225801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.225762    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:08:21.226880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.226852    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" containerName="payload-processing"
Nov 02 08:08:21.226880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.226879    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" containerName="payload-processing"
Nov 02 08:08:21.227042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.226948    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" containerName="payload-processing"
Nov 02 08:08:21.227042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.226958    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" containerName="payload-processing"
Nov 02 08:08:21.227145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.227130    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c909e69a-b40e-4dbb-b5a9-c78a5ec0c0c3" containerName="payload-processing"
Nov 02 08:08:21.227185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.227162    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc4a1b3-4580-4b6a-a79f-3d2efc0f7029" containerName="payload-processing"
Nov 02 08:08:21.232824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.232798    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.248700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.248672    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:08:21.256244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.256216    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"agc-cred-1ba7f0d22417\""
Nov 02 08:08:21.256562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.256541    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"routing-overlay\""
Nov 02 08:08:21.256562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.256540    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f2e06\""
Nov 02 08:08:21.256736 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.256573    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-e2e--9b11fa18-dockercfg-nznxp\""
Nov 02 08:08:21.260629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.260607    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"openshift-service-ca.crt\""
Nov 02 08:08:21.260732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.260626    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"kube-root-ca.crt\""
Nov 02 08:08:21.491514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.491407    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnz4\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.491715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.491673    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.491787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.491740    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.491787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.491769    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.611535 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.610805    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.611926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.611903    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.612143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.612123    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.612382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.612366    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnz4\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.614030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.613258    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.614208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.614151    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.615825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.615795    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.636375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.636345    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnz4\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:21.849549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:21.849507    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:22.160005 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:08:22.159970    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358c7584_55d1_4771_aa9a_0248bcdb0653.slice/crio-2a1b7ad2127140fc50fac07b975102aa2e074b91e274dfb87ac6cc298528b8b8 WatchSource:0}: Error finding container 2a1b7ad2127140fc50fac07b975102aa2e074b91e274dfb87ac6cc298528b8b8: Status 404 returned error can't find the container with id 2a1b7ad2127140fc50fac07b975102aa2e074b91e274dfb87ac6cc298528b8b8
Nov 02 08:08:22.162919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:22.162891    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:08:22.530691 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:22.530639    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" event={"ID":"358c7584-55d1-4771-aa9a-0248bcdb0653","Type":"ContainerStarted","Data":"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"}
Nov 02 08:08:22.531103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:22.530714    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" event={"ID":"358c7584-55d1-4771-aa9a-0248bcdb0653","Type":"ContainerStarted","Data":"2a1b7ad2127140fc50fac07b975102aa2e074b91e274dfb87ac6cc298528b8b8"}
Nov 02 08:08:22.531103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:22.530849    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:22.574273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:22.574214    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" podStartSLOduration=1.5741960640000001 podStartE2EDuration="1.574196064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:22.572260957 +0000 UTC m=+2647.334697868" watchObservedRunningTime="2026-11-02 08:08:22.574196064 +0000 UTC m=+2647.336632970"
Nov 02 08:08:27.555761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:27.555725    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:08:32.556208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.556166    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" podUID="98915278-0a84-4b1c-9456-170ce6456b1b" containerName="payload-processing" containerID="cri-o://1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9" gracePeriod=25
Nov 02 08:08:32.834984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.834951    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:08:32.978620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.978584    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\") pod \"98915278-0a84-4b1c-9456-170ce6456b1b\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") "
Nov 02 08:08:32.978848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.978693    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\") pod \"98915278-0a84-4b1c-9456-170ce6456b1b\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") "
Nov 02 08:08:32.978848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.978812    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\") pod \"98915278-0a84-4b1c-9456-170ce6456b1b\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") "
Nov 02 08:08:32.979255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.978881    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\") pod \"98915278-0a84-4b1c-9456-170ce6456b1b\" (UID: \"98915278-0a84-4b1c-9456-170ce6456b1b\") "
Nov 02 08:08:32.979255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.979212    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume" pod "98915278-0a84-4b1c-9456-170ce6456b1b" (UID: "98915278-0a84-4b1c-9456-170ce6456b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:32.979441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.979413    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume" pod "98915278-0a84-4b1c-9456-170ce6456b1b" (UID: "98915278-0a84-4b1c-9456-170ce6456b1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:32.979552 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.979535    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:32.979594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.979555    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98915278-0a84-4b1c-9456-170ce6456b1b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:32.981206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.981184    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh" pod "98915278-0a84-4b1c-9456-170ce6456b1b" (UID: "98915278-0a84-4b1c-9456-170ce6456b1b"). InnerVolumeSpecName "kube-api-access-86chh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:32.981342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:32.981253    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials" pod "98915278-0a84-4b1c-9456-170ce6456b1b" (UID: "98915278-0a84-4b1c-9456-170ce6456b1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:33.080829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.080744    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86chh\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-kube-api-access-86chh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:33.080829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.080773    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98915278-0a84-4b1c-9456-170ce6456b1b-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:33.541319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.541276    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:08:33.675126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.675035    4515 generic.go:356] "Generic (PLEG): container finished" podID="98915278-0a84-4b1c-9456-170ce6456b1b" containerID="1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9" exitCode=0
Nov 02 08:08:33.675126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.675083    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"
Nov 02 08:08:33.675126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.675107    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" event={"ID":"98915278-0a84-4b1c-9456-170ce6456b1b","Type":"ContainerDied","Data":"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"}
Nov 02 08:08:33.675655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.675157    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4" event={"ID":"98915278-0a84-4b1c-9456-170ce6456b1b","Type":"ContainerDied","Data":"4fe9512104ab9b37c4dea76a41c5fee18e73d3490ed322d49a61b6b99cccd146"}
Nov 02 08:08:33.675655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.675180    4515 scope.go:122] "RemoveContainer" containerID="1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"
Nov 02 08:08:33.685840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.685823    4515 scope.go:122] "RemoveContainer" containerID="1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"
Nov 02 08:08:33.686125 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:08:33.686100    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9\": container with ID starting with 1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9 not found: ID does not exist" containerID="1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"
Nov 02 08:08:33.686212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.686134    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9"} err="failed to get container status \"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9\": rpc error: code = NotFound desc = could not find container \"1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9\": container with ID starting with 1d07d9c93903534ad46ed7bafe414e5f6e29b6e6f47e86fb801463bad5838fb9 not found: ID does not exist"
Nov 02 08:08:33.738021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.737949    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:08:33.743566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:33.743539    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bc5gnjj4"]
Nov 02 08:08:35.374566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:35.374535    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98915278-0a84-4b1c-9456-170ce6456b1b" path="/var/lib/kubelet/pods/98915278-0a84-4b1c-9456-170ce6456b1b/volumes"
Nov 02 08:08:35.505915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:35.505875    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 08:08:35.506196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:35.506164    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy" containerID="cri-o://28135d849d64281d9eed4abe0fbb282b84068b22f2326e375679d0b16c3e126e" gracePeriod=30
Nov 02 08:08:40.731610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.731570    4515 generic.go:356] "Generic (PLEG): container finished" podID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerID="28135d849d64281d9eed4abe0fbb282b84068b22f2326e375679d0b16c3e126e" exitCode=0
Nov 02 08:08:40.732054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.731718    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" event={"ID":"aef95c73-f6ae-4db8-8987-00d772d9cd4c","Type":"ContainerDied","Data":"28135d849d64281d9eed4abe0fbb282b84068b22f2326e375679d0b16c3e126e"}
Nov 02 08:08:40.827338 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.827278    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 08:08:40.922878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.922833    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.923088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.922938    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.923088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923040    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.923391 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923359    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:08:40.923557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923495    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.923639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923594    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.923729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923664    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.924015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.923984    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:08:40.924133 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.924097    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:08:40.924200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.924174    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:40.924494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.924471    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.924590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.924540    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.924933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.924909    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\") pod \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\" (UID: \"aef95c73-f6ae-4db8-8987-00d772d9cd4c\") "
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.925420    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.925591    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926192    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926215    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926227    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926238    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926259    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926271    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:40.926802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.926657    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "kube-api-access-tm6hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:40.927217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.927192    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:08:40.928750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:40.928722    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token" pod "aef95c73-f6ae-4db8-8987-00d772d9cd4c" (UID: "aef95c73-f6ae-4db8-8987-00d772d9cd4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:41.028013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.027982    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-kube-api-access-tm6hl\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:41.028013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.028009    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:41.028013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.028018    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef95c73-f6ae-4db8-8987-00d772d9cd4c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:08:41.742441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.742350    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc" event={"ID":"aef95c73-f6ae-4db8-8987-00d772d9cd4c","Type":"ContainerDied","Data":"b27cef6b2e1393015f2cd394f14ef8e7173e845a1977697d256bd42d6a283a5d"}
Nov 02 08:08:41.742441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.742409    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"
Nov 02 08:08:41.742441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.742425    4515 scope.go:122] "RemoveContainer" containerID="28135d849d64281d9eed4abe0fbb282b84068b22f2326e375679d0b16c3e126e"
Nov 02 08:08:41.809691 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.809658    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 08:08:41.822500 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:41.822466    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-v52mc"]
Nov 02 08:08:43.374880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:43.374843    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" path="/var/lib/kubelet/pods/aef95c73-f6ae-4db8-8987-00d772d9cd4c/volumes"
Nov 02 08:08:47.724187 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:08:47.724146    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:08:57.952770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.952739    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:08:57.953470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953454    4515 cpu_manager.go:414] "containerMap: removing container" podUID="98915278-0a84-4b1c-9456-170ce6456b1b" containerName="payload-processing"
Nov 02 08:08:57.953470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953471    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98915278-0a84-4b1c-9456-170ce6456b1b" containerName="payload-processing"
Nov 02 08:08:57.953563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953507    4515 cpu_manager.go:414] "containerMap: removing container" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy"
Nov 02 08:08:57.953563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953512    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy"
Nov 02 08:08:57.953630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953606    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="98915278-0a84-4b1c-9456-170ce6456b1b" containerName="payload-processing"
Nov 02 08:08:57.953630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.953621    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef95c73-f6ae-4db8-8987-00d772d9cd4c" containerName="istio-proxy"
Nov 02 08:08:57.957434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.957414    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:57.972979 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.972950    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"e2e-cfl-4704f3-provider-cred\""
Nov 02 08:08:57.976234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.976210    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:08:57.985350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.984750    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-kmjj5\""
Nov 02 08:08:57.985350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.984776    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:08:57.985350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.984852    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:08:57.985613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.985560    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:08:57.985613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:57.985581    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:08:58.191339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.189204    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.191339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.189357    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.191339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.189488    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk27s\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.191339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.189623    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.290944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.290904    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk27s\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.291142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.291011    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.291142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.291052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.291142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.291119    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.291752 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.291725    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.291877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.291846    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.293767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.293739    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.314676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.314643    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk27s\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.572361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.572261    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:58.879002 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:08:58.878973    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e166b0_5bf1_48e5_bad0_e6215fa52607.slice/crio-936924faa3d4ae996754186dafa2c48e66eef0bc48f9ddc9cd814e4e0dc7238b WatchSource:0}: Error finding container 936924faa3d4ae996754186dafa2c48e66eef0bc48f9ddc9cd814e4e0dc7238b: Status 404 returned error can't find the container with id 936924faa3d4ae996754186dafa2c48e66eef0bc48f9ddc9cd814e4e0dc7238b
Nov 02 08:08:58.880060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:58.879986    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:08:59.869165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:59.869127    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" event={"ID":"69e166b0-5bf1-48e5-bad0-e6215fa52607","Type":"ContainerStarted","Data":"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"}
Nov 02 08:08:59.869690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:59.869212    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:08:59.869690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:59.869230    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" event={"ID":"69e166b0-5bf1-48e5-bad0-e6215fa52607","Type":"ContainerStarted","Data":"936924faa3d4ae996754186dafa2c48e66eef0bc48f9ddc9cd814e4e0dc7238b"}
Nov 02 08:08:59.912733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:08:59.912681    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" podStartSLOduration=2.91266799 podStartE2EDuration="2.91266799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:59.911365575 +0000 UTC m=+2684.673802480" watchObservedRunningTime="2026-11-02 08:08:59.91266799 +0000 UTC m=+2684.675104892"
Nov 02 08:09:05.901373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:05.901334    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:09:10.919071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:10.917588    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:09:11.040326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:11.040280    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:09:11.048821 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.048786    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/e2e-cfl-4704f3-provider-cred: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:11.048821 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.048818    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:11.049122 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.048924    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials podName:69e166b0-5bf1-48e5-bad0-e6215fa52607 nodeName:}" failed. No retries permitted until 2026-11-02 08:09:11.548902903 +0000 UTC m=+2696.311339810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607") : secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:11.557956 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.557914    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/e2e-cfl-4704f3-provider-cred: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:11.557956 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.557945    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:11.558227 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:11.558007    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials podName:69e166b0-5bf1-48e5-bad0-e6215fa52607 nodeName:}" failed. No retries permitted until 2026-11-02 08:09:12.557986135 +0000 UTC m=+2697.320423019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607") : secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:12.574961 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:12.574927    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/e2e-cfl-4704f3-provider-cred: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:12.574961 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:12.574956    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:12.575471 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:12.575024    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials podName:69e166b0-5bf1-48e5-bad0-e6215fa52607 nodeName:}" failed. No retries permitted until 2026-11-02 08:09:14.575005286 +0000 UTC m=+2699.337442171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607") : secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:14.605837 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:14.605804    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-736f7c/e2e-cfl-4704f3-provider-cred: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:14.605837 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:14.605834    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf: secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:14.606297 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:14.605901    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials podName:69e166b0-5bf1-48e5-bad0-e6215fa52607 nodeName:}" failed. No retries permitted until 2026-11-02 08:09:18.605880347 +0000 UTC m=+2703.368317235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607") : secret "e2e-cfl-4704f3-provider-cred" not found
Nov 02 08:09:16.040884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.040813    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" podUID="69e166b0-5bf1-48e5-bad0-e6215fa52607" containerName="payload-processing" containerID="cri-o://2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b" gracePeriod=25
Nov 02 08:09:16.326300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.326267    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:09:16.433900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.433859    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\") pod \"69e166b0-5bf1-48e5-bad0-e6215fa52607\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") "
Nov 02 08:09:16.434107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.433973    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\") pod \"69e166b0-5bf1-48e5-bad0-e6215fa52607\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") "
Nov 02 08:09:16.434176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434105    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\") pod \"69e166b0-5bf1-48e5-bad0-e6215fa52607\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") "
Nov 02 08:09:16.434393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434374    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\") pod \"69e166b0-5bf1-48e5-bad0-e6215fa52607\" (UID: \"69e166b0-5bf1-48e5-bad0-e6215fa52607\") "
Nov 02 08:09:16.434491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434420    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume" pod "69e166b0-5bf1-48e5-bad0-e6215fa52607" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:16.434813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434787    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume" pod "69e166b0-5bf1-48e5-bad0-e6215fa52607" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:16.434971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434954    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.435014 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.434978    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e166b0-5bf1-48e5-bad0-e6215fa52607-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.436285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.436262    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials" pod "69e166b0-5bf1-48e5-bad0-e6215fa52607" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:16.436416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.436400    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s" pod "69e166b0-5bf1-48e5-bad0-e6215fa52607" (UID: "69e166b0-5bf1-48e5-bad0-e6215fa52607"). InnerVolumeSpecName "kube-api-access-dk27s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:16.536658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.536624    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk27s\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-kube-api-access-dk27s\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.536658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:16.536651    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69e166b0-5bf1-48e5-bad0-e6215fa52607-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:17.019109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.019063    4515 generic.go:356] "Generic (PLEG): container finished" podID="69e166b0-5bf1-48e5-bad0-e6215fa52607" containerID="2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b" exitCode=0
Nov 02 08:09:17.019346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.019131    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"
Nov 02 08:09:17.019346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.019149    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" event={"ID":"69e166b0-5bf1-48e5-bad0-e6215fa52607","Type":"ContainerDied","Data":"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"}
Nov 02 08:09:17.019346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.019242    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf" event={"ID":"69e166b0-5bf1-48e5-bad0-e6215fa52607","Type":"ContainerDied","Data":"936924faa3d4ae996754186dafa2c48e66eef0bc48f9ddc9cd814e4e0dc7238b"}
Nov 02 08:09:17.019346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.019273    4515 scope.go:122] "RemoveContainer" containerID="2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"
Nov 02 08:09:17.030568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.030549    4515 scope.go:122] "RemoveContainer" containerID="2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"
Nov 02 08:09:17.030873 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:17.030849    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b\": container with ID starting with 2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b not found: ID does not exist" containerID="2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"
Nov 02 08:09:17.030923 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.030882    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b"} err="failed to get container status \"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b\": rpc error: code = NotFound desc = could not find container \"2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b\": container with ID starting with 2676470da2f81f8a36bea974e65c72fddd2b756a7e915b7012bcee81e706ef6b not found: ID does not exist"
Nov 02 08:09:17.104914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.104879    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:09:17.163331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.158489    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdbcbdf"]
Nov 02 08:09:17.374461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:17.374376    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e166b0-5bf1-48e5-bad0-e6215fa52607" path="/var/lib/kubelet/pods/69e166b0-5bf1-48e5-bad0-e6215fa52607/volumes"
Nov 02 08:09:17.724263 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:17.724174    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:09:27.001742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.001709    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:09:27.002451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.002435    4515 cpu_manager.go:414] "containerMap: removing container" podUID="69e166b0-5bf1-48e5-bad0-e6215fa52607" containerName="payload-processing"
Nov 02 08:09:27.002510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.002452    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e166b0-5bf1-48e5-bad0-e6215fa52607" containerName="payload-processing"
Nov 02 08:09:27.002579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.002570    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e166b0-5bf1-48e5-bad0-e6215fa52607" containerName="payload-processing"
Nov 02 08:09:27.007989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.007969    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.027697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.027651    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-247a3d77fa68\""
Nov 02 08:09:27.029261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.029234    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 08:09:27.030389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.030366    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:09:27.034553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.034532    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 08:09:27.035109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.035088    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l2lh7\""
Nov 02 08:09:27.035245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.035168    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 08:09:27.035362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.035246    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 08:09:27.126500 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.126459    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.126713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.126689    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.126860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.126835    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7mh\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.127023 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.127002    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.231329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.231205    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.231329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.231275    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.231611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.231449    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.231611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.231550    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7mh\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.234382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.232862    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.258374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.237994    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.275587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.260149    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7mh\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.275587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.267354    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\") pod \"payload-processing-external-model-5649fd4f9b-jj5dj\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") " pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.323321 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.323183    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:27.640747 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:09:27.640719    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode822946a_4fe4_4c92_8fdd_048e568d2be9.slice/crio-523e8cf35c10ec65d977586fce56595158d0a3803199f566d048bc23c771fc0a WatchSource:0}: Error finding container 523e8cf35c10ec65d977586fce56595158d0a3803199f566d048bc23c771fc0a: Status 404 returned error can't find the container with id 523e8cf35c10ec65d977586fce56595158d0a3803199f566d048bc23c771fc0a
Nov 02 08:09:27.641805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:27.641777    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:09:28.111236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:28.111197    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" event={"ID":"e822946a-4fe4-4c92-8fdd-048e568d2be9","Type":"ContainerStarted","Data":"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"}
Nov 02 08:09:28.111697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:28.111257    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" event={"ID":"e822946a-4fe4-4c92-8fdd-048e568d2be9","Type":"ContainerStarted","Data":"523e8cf35c10ec65d977586fce56595158d0a3803199f566d048bc23c771fc0a"}
Nov 02 08:09:28.111697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:28.111285    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:28.158366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:28.158297    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" podStartSLOduration=2.158282516 podStartE2EDuration="2.158282516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:28.15495147 +0000 UTC m=+2712.917388376" watchObservedRunningTime="2026-11-02 08:09:28.158282516 +0000 UTC m=+2712.920719470"
Nov 02 08:09:32.083059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.081850    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:32.103821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.103754    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:32.104013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.103837    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.120105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.120077    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-a8ddd0a5d184\""
Nov 02 08:09:32.132639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.132579    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"routing-overlay\""
Nov 02 08:09:32.132639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.132589    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"openshift-service-ca.crt\""
Nov 02 08:09:32.132639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.132587    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-e2e--c1e7f36e-dockercfg-stf7d\""
Nov 02 08:09:32.132880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.132700    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:09:32.132880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.132780    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"kube-root-ca.crt\""
Nov 02 08:09:32.213862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.213828    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdl2\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.214044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.213990    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.214044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.214028    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.214201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.214183    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.316098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.316057    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdl2\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.316374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.316187    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.316374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.316211    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.316374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.316287    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.317118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.317094    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.317214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.317126    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.319133 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.319095    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.380551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.380481    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdl2\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.428191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.428158    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:32.748008 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:09:32.747978    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876ccfb6_ce93_4e63_9bbb_403fa84745ee.slice/crio-67a1a2a808938928c278af54bc9cccfb33c8a079fe0e68fd9276dcccb033db88 WatchSource:0}: Error finding container 67a1a2a808938928c278af54bc9cccfb33c8a079fe0e68fd9276dcccb033db88: Status 404 returned error can't find the container with id 67a1a2a808938928c278af54bc9cccfb33c8a079fe0e68fd9276dcccb033db88
Nov 02 08:09:32.749634 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.749610    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:32.765898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:32.765864    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:33.213331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:33.209965    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" event={"ID":"876ccfb6-ce93-4e63-9bbb-403fa84745ee","Type":"ContainerStarted","Data":"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"}
Nov 02 08:09:33.213331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:33.210034    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" event={"ID":"876ccfb6-ce93-4e63-9bbb-403fa84745ee","Type":"ContainerStarted","Data":"67a1a2a808938928c278af54bc9cccfb33c8a079fe0e68fd9276dcccb033db88"}
Nov 02 08:09:33.213331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:33.210217    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:33.292000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:33.284910    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" podStartSLOduration=1.284889174 podStartE2EDuration="1.284889174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:33.281233636 +0000 UTC m=+2718.043670543" watchObservedRunningTime="2026-11-02 08:09:33.284889174 +0000 UTC m=+2718.047326083"
Nov 02 08:09:38.210878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.210811    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" podUID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" containerName="payload-processing" containerID="cri-o://32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5" gracePeriod=25
Nov 02 08:09:38.501799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.501772    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:38.564365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564299    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\") pod \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") "
Nov 02 08:09:38.564538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564434    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\") pod \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") "
Nov 02 08:09:38.564620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564541    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\") pod \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") "
Nov 02 08:09:38.564696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564679    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\") pod \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\" (UID: \"876ccfb6-ce93-4e63-9bbb-403fa84745ee\") "
Nov 02 08:09:38.564768 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564723    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume" pod "876ccfb6-ce93-4e63-9bbb-403fa84745ee" (UID: "876ccfb6-ce93-4e63-9bbb-403fa84745ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:38.564887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.564862    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume" pod "876ccfb6-ce93-4e63-9bbb-403fa84745ee" (UID: "876ccfb6-ce93-4e63-9bbb-403fa84745ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:38.565760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.565729    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.565760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.565762    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876ccfb6-ce93-4e63-9bbb-403fa84745ee-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.566856 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.566834    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials" pod "876ccfb6-ce93-4e63-9bbb-403fa84745ee" (UID: "876ccfb6-ce93-4e63-9bbb-403fa84745ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:38.566987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.566961    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2" pod "876ccfb6-ce93-4e63-9bbb-403fa84745ee" (UID: "876ccfb6-ce93-4e63-9bbb-403fa84745ee"). InnerVolumeSpecName "kube-api-access-wjdl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:38.667486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.667445    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjdl2\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-kube-api-access-wjdl2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.667486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:38.667476    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/876ccfb6-ce93-4e63-9bbb-403fa84745ee-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:39.120954 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.120924    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:09:39.251759 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.251726    4515 generic.go:356] "Generic (PLEG): container finished" podID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" containerID="32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5" exitCode=0
Nov 02 08:09:39.252178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.251768    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"
Nov 02 08:09:39.252178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.251799    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" event={"ID":"876ccfb6-ce93-4e63-9bbb-403fa84745ee","Type":"ContainerDied","Data":"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"}
Nov 02 08:09:39.252178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.251853    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7" event={"ID":"876ccfb6-ce93-4e63-9bbb-403fa84745ee","Type":"ContainerDied","Data":"67a1a2a808938928c278af54bc9cccfb33c8a079fe0e68fd9276dcccb033db88"}
Nov 02 08:09:39.252178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.251877    4515 scope.go:122] "RemoveContainer" containerID="32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"
Nov 02 08:09:39.262792 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.262773    4515 scope.go:122] "RemoveContainer" containerID="32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"
Nov 02 08:09:39.263071 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:39.263050    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5\": container with ID starting with 32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5 not found: ID does not exist" containerID="32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"
Nov 02 08:09:39.263120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.263081    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5"} err="failed to get container status \"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5\": rpc error: code = NotFound desc = could not find container \"32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5\": container with ID starting with 32331b448846199e92b5c354c95ab5593031b447cad20bd020866f8abdc576a5 not found: ID does not exist"
Nov 02 08:09:39.315976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.315935    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:39.321676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.321648    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6b294s7"]
Nov 02 08:09:39.375788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:39.375713    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" path="/var/lib/kubelet/pods/876ccfb6-ce93-4e63-9bbb-403fa84745ee/volumes"
Nov 02 08:09:45.983188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:45.983148    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:09:47.725265 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:47.725221    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:09:50.983940 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:50.983895    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" podUID="358c7584-55d1-4771-aa9a-0248bcdb0653" containerName="payload-processing" containerID="cri-o://15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441" gracePeriod=25
Nov 02 08:09:51.274886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.274854    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:09:51.378795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.378762    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\") pod \"358c7584-55d1-4771-aa9a-0248bcdb0653\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") "
Nov 02 08:09:51.378969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.378841    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\") pod \"358c7584-55d1-4771-aa9a-0248bcdb0653\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") "
Nov 02 08:09:51.378969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.378927    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\") pod \"358c7584-55d1-4771-aa9a-0248bcdb0653\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") "
Nov 02 08:09:51.379065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.379011    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\") pod \"358c7584-55d1-4771-aa9a-0248bcdb0653\" (UID: \"358c7584-55d1-4771-aa9a-0248bcdb0653\") "
Nov 02 08:09:51.379323 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.379282    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume" pod "358c7584-55d1-4771-aa9a-0248bcdb0653" (UID: "358c7584-55d1-4771-aa9a-0248bcdb0653"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:51.379475 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.379454    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume" pod "358c7584-55d1-4771-aa9a-0248bcdb0653" (UID: "358c7584-55d1-4771-aa9a-0248bcdb0653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:51.379737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.379714    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:51.379828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.379742    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358c7584-55d1-4771-aa9a-0248bcdb0653-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:51.381225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.381205    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials" pod "358c7584-55d1-4771-aa9a-0248bcdb0653" (UID: "358c7584-55d1-4771-aa9a-0248bcdb0653"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:51.381343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.381324    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4" pod "358c7584-55d1-4771-aa9a-0248bcdb0653" (UID: "358c7584-55d1-4771-aa9a-0248bcdb0653"). InnerVolumeSpecName "kube-api-access-8nnz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:51.426010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.425974    4515 generic.go:356] "Generic (PLEG): container finished" podID="358c7584-55d1-4771-aa9a-0248bcdb0653" containerID="15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441" exitCode=0
Nov 02 08:09:51.426182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.426029    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"
Nov 02 08:09:51.426182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.426070    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" event={"ID":"358c7584-55d1-4771-aa9a-0248bcdb0653","Type":"ContainerDied","Data":"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"}
Nov 02 08:09:51.426182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.426133    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5" event={"ID":"358c7584-55d1-4771-aa9a-0248bcdb0653","Type":"ContainerDied","Data":"2a1b7ad2127140fc50fac07b975102aa2e074b91e274dfb87ac6cc298528b8b8"}
Nov 02 08:09:51.426182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.426163    4515 scope.go:122] "RemoveContainer" containerID="15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"
Nov 02 08:09:51.436829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.436811    4515 scope.go:122] "RemoveContainer" containerID="15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"
Nov 02 08:09:51.437127 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:09:51.437106    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441\": container with ID starting with 15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441 not found: ID does not exist" containerID="15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"
Nov 02 08:09:51.437202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.437136    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441"} err="failed to get container status \"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441\": rpc error: code = NotFound desc = could not find container \"15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441\": container with ID starting with 15ab436afa0a51a4547e38a201d42178d909189f1becc367ba999751df83f441 not found: ID does not exist"
Nov 02 08:09:51.480805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.480771    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnz4\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-kube-api-access-8nnz4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:51.480805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.480803    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/358c7584-55d1-4771-aa9a-0248bcdb0653-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:09:51.485417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.485387    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:09:51.490451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:51.490427    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-6744hmhq5"]
Nov 02 08:09:53.374290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:53.374254    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358c7584-55d1-4771-aa9a-0248bcdb0653" path="/var/lib/kubelet/pods/358c7584-55d1-4771-aa9a-0248bcdb0653/volumes"
Nov 02 08:09:57.964545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:09:57.964455    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:10:05.925652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.925616    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:05.926335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926320    4515 cpu_manager.go:414] "containerMap: removing container" podUID="358c7584-55d1-4771-aa9a-0248bcdb0653" containerName="payload-processing"
Nov 02 08:10:05.926335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926335    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358c7584-55d1-4771-aa9a-0248bcdb0653" containerName="payload-processing"
Nov 02 08:10:05.926426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926355    4515 cpu_manager.go:414] "containerMap: removing container" podUID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" containerName="payload-processing"
Nov 02 08:10:05.926426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926360    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" containerName="payload-processing"
Nov 02 08:10:05.926492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926463    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="358c7584-55d1-4771-aa9a-0248bcdb0653" containerName="payload-processing"
Nov 02 08:10:05.926492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.926475    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="876ccfb6-ce93-4e63-9bbb-403fa84745ee" containerName="payload-processing"
Nov 02 08:10:05.931557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.931534    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:05.950702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.950677    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-212653738a3c\""
Nov 02 08:10:05.963461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.963429    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-621795298e14\""
Nov 02 08:10:05.967681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:05.967656    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:06.013249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.013213    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrn2\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.013456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.013268    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.013456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.013431    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.013588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.013489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.027594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.027567    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-81cc97ec1564\""
Nov 02 08:10:06.114844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.114805    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrn2\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.114844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.114850    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.115079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.114902    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.115079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.114934    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.115637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.115607    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.115864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.115841    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.117763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.117742    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.140537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.140513    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrn2\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\") pod \"payload-processing-external-model-5b8468948b-9l7j4\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") " pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.246970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.246571    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.564508 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:10:06.564476    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fc9e8b_b558_42cb_a85e_bbe54992b1e8.slice/crio-cc7f5db4d24b0ad184a726655f1d14a948ee5359177a1bd5d64b0381eaa2c268 WatchSource:0}: Error finding container cc7f5db4d24b0ad184a726655f1d14a948ee5359177a1bd5d64b0381eaa2c268: Status 404 returned error can't find the container with id cc7f5db4d24b0ad184a726655f1d14a948ee5359177a1bd5d64b0381eaa2c268
Nov 02 08:10:06.565947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.565919    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:06.735118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.735080    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" event={"ID":"70fc9e8b-b558-42cb-a85e-bbe54992b1e8","Type":"ContainerStarted","Data":"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"}
Nov 02 08:10:06.735300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.735130    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" event={"ID":"70fc9e8b-b558-42cb-a85e-bbe54992b1e8","Type":"ContainerStarted","Data":"cc7f5db4d24b0ad184a726655f1d14a948ee5359177a1bd5d64b0381eaa2c268"}
Nov 02 08:10:06.735379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.735325    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:06.783682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:06.783628    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" podStartSLOduration=1.783613278 podStartE2EDuration="1.783613278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:06.779913539 +0000 UTC m=+2751.542350444" watchObservedRunningTime="2026-11-02 08:10:06.783613278 +0000 UTC m=+2751.546050183"
Nov 02 08:10:12.071682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:12.071641    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:12.075195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:12.075167    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:17.071912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.071872    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" podUID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" containerName="payload-processing" containerID="cri-o://76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4" gracePeriod=25
Nov 02 08:10:17.395461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.395434    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:17.482961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.482914    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\") pod \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") "
Nov 02 08:10:17.482961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.482953    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\") pod \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") "
Nov 02 08:10:17.483245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.482981    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\") pod \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") "
Nov 02 08:10:17.483245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.483194    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\") pod \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\" (UID: \"70fc9e8b-b558-42cb-a85e-bbe54992b1e8\") "
Nov 02 08:10:17.483441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.483378    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume" pod "70fc9e8b-b558-42cb-a85e-bbe54992b1e8" (UID: "70fc9e8b-b558-42cb-a85e-bbe54992b1e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:17.483652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.483625    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume" pod "70fc9e8b-b558-42cb-a85e-bbe54992b1e8" (UID: "70fc9e8b-b558-42cb-a85e-bbe54992b1e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:17.483978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.483958    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:17.484082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.483984    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:17.485300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.485273    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials" pod "70fc9e8b-b558-42cb-a85e-bbe54992b1e8" (UID: "70fc9e8b-b558-42cb-a85e-bbe54992b1e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:17.485300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.485287    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2" pod "70fc9e8b-b558-42cb-a85e-bbe54992b1e8" (UID: "70fc9e8b-b558-42cb-a85e-bbe54992b1e8"). InnerVolumeSpecName "kube-api-access-6rrn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:17.585090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.585051    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rrn2\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-kube-api-access-6rrn2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:17.585090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.585089    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70fc9e8b-b558-42cb-a85e-bbe54992b1e8-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:17.724738 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:10:17.724643    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:10:17.884952 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.884921    4515 generic.go:356] "Generic (PLEG): container finished" podID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" containerID="76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4" exitCode=0
Nov 02 08:10:17.885135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.884982    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"
Nov 02 08:10:17.885135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.884998    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" event={"ID":"70fc9e8b-b558-42cb-a85e-bbe54992b1e8","Type":"ContainerDied","Data":"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"}
Nov 02 08:10:17.885135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.885082    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4" event={"ID":"70fc9e8b-b558-42cb-a85e-bbe54992b1e8","Type":"ContainerDied","Data":"cc7f5db4d24b0ad184a726655f1d14a948ee5359177a1bd5d64b0381eaa2c268"}
Nov 02 08:10:17.885135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.885112    4515 scope.go:122] "RemoveContainer" containerID="76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"
Nov 02 08:10:17.896508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.896480    4515 scope.go:122] "RemoveContainer" containerID="76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"
Nov 02 08:10:17.896804 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:10:17.896783    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4\": container with ID starting with 76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4 not found: ID does not exist" containerID="76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"
Nov 02 08:10:17.896884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.896817    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4"} err="failed to get container status \"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4\": rpc error: code = NotFound desc = could not find container \"76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4\": container with ID starting with 76eaa13444aab50acfc4dddfc6047a9cf76e3d072d0e71dda70263c1d6377ef4 not found: ID does not exist"
Nov 02 08:10:17.943249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.943225    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:17.955922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:17.955893    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8468948b-9l7j4"]
Nov 02 08:10:19.375300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:19.375263    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" path="/var/lib/kubelet/pods/70fc9e8b-b558-42cb-a85e-bbe54992b1e8/volumes"
Nov 02 08:10:22.262174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.262136    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:10:22.262973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.262953    4515 cpu_manager.go:414] "containerMap: removing container" podUID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" containerName="payload-processing"
Nov 02 08:10:22.262973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.262974    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" containerName="payload-processing"
Nov 02 08:10:22.263162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.263082    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fc9e8b-b558-42cb-a85e-bbe54992b1e8" containerName="payload-processing"
Nov 02 08:10:22.268612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.268589    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.285558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.285525    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-212653738a3c\""
Nov 02 08:10:22.288370 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.288344    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:10:22.370399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.370368    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j89\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.370605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.370512    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.370689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.370655    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.370736 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.370699    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.385291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.385259    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-621795298e14\""
Nov 02 08:10:22.472597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.472560    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j89\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.472788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.472760    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.473026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.472988    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.473166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.473031    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.473680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.473642    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.474107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.474084    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.475812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.475790    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.498555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.498523    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j89\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\") pod \"payload-processing-external-model-744487595b-p774w\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") " pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.625033 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.624947    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:22.984359 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:10:22.984323    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c654f3_32e4_4a94_88ff_186098c11311.slice/crio-03cab93447a8d2325630c2369a1a34bb1769cde3a0335cbbcf84f5e8a59ef821 WatchSource:0}: Error finding container 03cab93447a8d2325630c2369a1a34bb1769cde3a0335cbbcf84f5e8a59ef821: Status 404 returned error can't find the container with id 03cab93447a8d2325630c2369a1a34bb1769cde3a0335cbbcf84f5e8a59ef821
Nov 02 08:10:22.985808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:22.985779    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:10:23.815121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.814822    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:10:23.826554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.826528    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:23.846253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.846223    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:10:23.927477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.927441    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" event={"ID":"e7c654f3-32e4-4a94-88ff-186098c11311","Type":"ContainerStarted","Data":"5824a98f7d3052fd2ce73bfc0b2109c2f970224b1569fdbbf5c093faa266709f"}
Nov 02 08:10:23.927663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.927501    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" event={"ID":"e7c654f3-32e4-4a94-88ff-186098c11311","Type":"ContainerStarted","Data":"03cab93447a8d2325630c2369a1a34bb1769cde3a0335cbbcf84f5e8a59ef821"}
Nov 02 08:10:23.927663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:23.927576    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:24.009354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009281    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009461    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009495    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009519    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009587    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009638    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009687    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tr9v\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.009846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.009774    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.110909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.110824    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.110909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.110882    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.110909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.110910    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.110935    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.110993    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111035    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111064    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tr9v\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111168    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111296    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111360    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111369    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.111896 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.111866    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.112069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.112044    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.112207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.112113    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.113474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.113449    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.114003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.113983    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.137671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.137633    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tr9v\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.139203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.139179    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ckx7l\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.150220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.150195    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:24.475526 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:10:24.475497    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e276e2_c6f8_4fad_9c35_f898381be2de.slice/crio-ba5ddc5c9217641e859649ea3d0b296e7fb8937a6ca22800e9ce30f7e3259e3c WatchSource:0}: Error finding container ba5ddc5c9217641e859649ea3d0b296e7fb8937a6ca22800e9ce30f7e3259e3c: Status 404 returned error can't find the container with id ba5ddc5c9217641e859649ea3d0b296e7fb8937a6ca22800e9ce30f7e3259e3c
Nov 02 08:10:24.475901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.475843    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" podStartSLOduration=2.475823503 podStartE2EDuration="2.475823503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:24.01140629 +0000 UTC m=+2768.773843197" watchObservedRunningTime="2026-11-02 08:10:24.475823503 +0000 UTC m=+2769.238260410"
Nov 02 08:10:24.478247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.478221    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:10:24.478887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.478849    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.478988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.478923    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.478988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.478960    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.999379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:24.999336    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" event={"ID":"03e276e2-c6f8-4fad-9c35-f898381be2de","Type":"ContainerStarted","Data":"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"}
Nov 02 08:10:25.000598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:25.000569    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" event={"ID":"03e276e2-c6f8-4fad-9c35-f898381be2de","Type":"ContainerStarted","Data":"ba5ddc5c9217641e859649ea3d0b296e7fb8937a6ca22800e9ce30f7e3259e3c"}
Nov 02 08:10:25.133541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:25.122151    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podStartSLOduration=2.12213161 podStartE2EDuration="2.12213161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:25.098008949 +0000 UTC m=+2769.860445857" watchObservedRunningTime="2026-11-02 08:10:25.12213161 +0000 UTC m=+2769.884568516"
Nov 02 08:10:25.173332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:25.162797    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:25.199335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:25.190550    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:25.199335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:25.190622    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:26.152332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:26.150762    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:26.152332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:26.150840    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:27.150951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:27.150911    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:27.151147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:27.150979    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:28.151676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:28.151635    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:28.152097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:28.151714    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:29.158332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:29.152677    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:29.158332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:29.152739    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:30.150890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:30.150852    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:30.151087 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:30.150915    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:31.151275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:31.151237    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:31.151711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:31.151296    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:32.150867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:32.150825    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ckx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 08:10:32.151066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:32.150896    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 08:10:33.526360 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:33.524777    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:33.526360 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:33.525702    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:33.527004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:33.526896    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:10:35.002948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:35.002918    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:10:35.100885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:35.100838    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:10:38.859496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.859462    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:10:38.864182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.864162    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890257    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890611    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890777 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890684    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890834    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890877    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890927    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.890982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.890962    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhzj\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.891090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.891032    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.891152 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.891093    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.896590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.896561    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:10:38.992092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992054    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992110    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992132    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhzj\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992231    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992350    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992432    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992462    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992605    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992716    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992715    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.992953 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992931    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.993004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.992958    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.993288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.993271    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.995192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.995168    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:38.995600 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:38.995577    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:39.072860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.072825    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:39.073186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.073162    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhzj\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gsh6q\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:39.182100 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.181997    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:39.514260 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:10:39.514229    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189a9882_e40a_4271_bf60_01f888c4e043.slice/crio-b32dd1b092fef056a4dc94712485e6180da544df88ad80ebb595abd73e01d9cc WatchSource:0}: Error finding container b32dd1b092fef056a4dc94712485e6180da544df88ad80ebb595abd73e01d9cc: Status 404 returned error can't find the container with id b32dd1b092fef056a4dc94712485e6180da544df88ad80ebb595abd73e01d9cc
Nov 02 08:10:39.515378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.515353    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:10:39.517806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.517769    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:39.517881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.517839    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:39.517881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:39.517867    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:40.102187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.102137    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" podUID="e822946a-4fe4-4c92-8fdd-048e568d2be9" containerName="payload-processing" containerID="cri-o://7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179" gracePeriod=25
Nov 02 08:10:40.146616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.146581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" event={"ID":"189a9882-e40a-4271-bf60-01f888c4e043","Type":"ContainerStarted","Data":"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"}
Nov 02 08:10:40.146804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.146643    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" event={"ID":"189a9882-e40a-4271-bf60-01f888c4e043","Type":"ContainerStarted","Data":"b32dd1b092fef056a4dc94712485e6180da544df88ad80ebb595abd73e01d9cc"}
Nov 02 08:10:40.182847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.182820    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:40.184713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.184687    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:40.184858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.184746    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:40.203669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.203612    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podStartSLOduration=2.203597734 podStartE2EDuration="2.203597734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:40.198956009 +0000 UTC m=+2784.961392916" watchObservedRunningTime="2026-11-02 08:10:40.203597734 +0000 UTC m=+2784.966034639"
Nov 02 08:10:40.417216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.417188    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:10:40.543703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.543666    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\") pod \"e822946a-4fe4-4c92-8fdd-048e568d2be9\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") "
Nov 02 08:10:40.544030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.544013    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\") pod \"e822946a-4fe4-4c92-8fdd-048e568d2be9\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") "
Nov 02 08:10:40.544170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.544156    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\") pod \"e822946a-4fe4-4c92-8fdd-048e568d2be9\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") "
Nov 02 08:10:40.544559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.544537    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\") pod \"e822946a-4fe4-4c92-8fdd-048e568d2be9\" (UID: \"e822946a-4fe4-4c92-8fdd-048e568d2be9\") "
Nov 02 08:10:40.545749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.545723    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume" pod "e822946a-4fe4-4c92-8fdd-048e568d2be9" (UID: "e822946a-4fe4-4c92-8fdd-048e568d2be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:40.548671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.548643    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume" pod "e822946a-4fe4-4c92-8fdd-048e568d2be9" (UID: "e822946a-4fe4-4c92-8fdd-048e568d2be9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:40.549627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.549601    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials" pod "e822946a-4fe4-4c92-8fdd-048e568d2be9" (UID: "e822946a-4fe4-4c92-8fdd-048e568d2be9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:40.549745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.549640    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh" pod "e822946a-4fe4-4c92-8fdd-048e568d2be9" (UID: "e822946a-4fe4-4c92-8fdd-048e568d2be9"). InnerVolumeSpecName "kube-api-access-tc7mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:40.647579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.647484    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc7mh\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-kube-api-access-tc7mh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:40.647579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.647528    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e822946a-4fe4-4c92-8fdd-048e568d2be9-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:40.647579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.647546    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:40.647579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:40.647559    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e822946a-4fe4-4c92-8fdd-048e568d2be9-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:10:41.156051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.156018    4515 generic.go:356] "Generic (PLEG): container finished" podID="e822946a-4fe4-4c92-8fdd-048e568d2be9" containerID="7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179" exitCode=0
Nov 02 08:10:41.156494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.156076    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"
Nov 02 08:10:41.156494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.156106    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" event={"ID":"e822946a-4fe4-4c92-8fdd-048e568d2be9","Type":"ContainerDied","Data":"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"}
Nov 02 08:10:41.156494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.156200    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj" event={"ID":"e822946a-4fe4-4c92-8fdd-048e568d2be9","Type":"ContainerDied","Data":"523e8cf35c10ec65d977586fce56595158d0a3803199f566d048bc23c771fc0a"}
Nov 02 08:10:41.156494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.156237    4515 scope.go:122] "RemoveContainer" containerID="7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"
Nov 02 08:10:41.168058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.168040    4515 scope.go:122] "RemoveContainer" containerID="7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"
Nov 02 08:10:41.168349 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:10:41.168330    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179\": container with ID starting with 7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179 not found: ID does not exist" containerID="7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"
Nov 02 08:10:41.168429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.168355    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179"} err="failed to get container status \"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179\": rpc error: code = NotFound desc = could not find container \"7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179\": container with ID starting with 7c2d1eccab191e0bf3c174f03bd82b4a069e80a3737a1dbbf9a6f0b5ae30f179 not found: ID does not exist"
Nov 02 08:10:41.182521 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.182496    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:41.182639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.182541    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:41.243669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.243623    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:10:41.247132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.247105    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5649fd4f9b-jj5dj"]
Nov 02 08:10:41.375331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:41.375287    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e822946a-4fe4-4c92-8fdd-048e568d2be9" path="/var/lib/kubelet/pods/e822946a-4fe4-4c92-8fdd-048e568d2be9/volumes"
Nov 02 08:10:42.198334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:42.197512    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:42.198334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:42.197575    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:43.182603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:43.182563    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:43.182832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:43.182618    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:44.183408 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:44.183375    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:44.184013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:44.183981    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:45.184702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:45.184654    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:45.185250 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:45.184726    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:45.315476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:45.315440    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:10:45.315641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:45.315514    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:10:46.183231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:46.183197    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:46.183411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:46.183255    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:47.182851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:47.182814    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gsh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 08:10:47.183358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:47.182891    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 08:10:47.724516 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:10:47.724480    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:10:48.707887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:48.707849    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:48.708497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:48.708349    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:10:48.709381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:10:48.709358    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:11:08.913118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.913074    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:11:08.913857 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.913840    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e822946a-4fe4-4c92-8fdd-048e568d2be9" containerName="payload-processing"
Nov 02 08:11:08.913857 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.913858    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e822946a-4fe4-4c92-8fdd-048e568d2be9" containerName="payload-processing"
Nov 02 08:11:08.914011 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.913999    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e822946a-4fe4-4c92-8fdd-048e568d2be9" containerName="payload-processing"
Nov 02 08:11:08.918964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.918944    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:08.943476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:08.943451    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:11:09.022451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022417    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022543    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022599    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdtc\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022638    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022745    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022837 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022790    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.022918 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022892    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.023048 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022952    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.023048 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.022989    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124643    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124746    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124773    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdtc\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124805    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124856    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.124888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.124875    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125030    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125044    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125178    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125209    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125229    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125432    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125520    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.125694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.125673    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.127166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.127145    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.127723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.127699    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.148619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.148591    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.148619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.148615    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdtc\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8qdvr\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.233981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.233900    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:09.556826 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:11:09.556797    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b641edf_1f18_4f23_a30e_5d177cb1633e.slice/crio-fdd495931a8d2904a92f22c3409d827d9dd6f59a0ba74e8d2dfa0b04aa296b0e WatchSource:0}: Error finding container fdd495931a8d2904a92f22c3409d827d9dd6f59a0ba74e8d2dfa0b04aa296b0e: Status 404 returned error can't find the container with id fdd495931a8d2904a92f22c3409d827d9dd6f59a0ba74e8d2dfa0b04aa296b0e
Nov 02 08:11:09.558278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.558251    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:11:09.559700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.559666    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:11:09.559788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.559731    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:11:09.559788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:09.559761    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:11:10.491847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:10.491812    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" event={"ID":"9b641edf-1f18-4f23-a30e-5d177cb1633e","Type":"ContainerStarted","Data":"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"}
Nov 02 08:11:10.492419 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:10.491864    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" event={"ID":"9b641edf-1f18-4f23-a30e-5d177cb1633e","Type":"ContainerStarted","Data":"fdd495931a8d2904a92f22c3409d827d9dd6f59a0ba74e8d2dfa0b04aa296b0e"}
Nov 02 08:11:10.569291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:10.569231    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podStartSLOduration=2.5692173560000002 podStartE2EDuration="2.569217356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:10.56675261 +0000 UTC m=+2815.329189659" watchObservedRunningTime="2026-11-02 08:11:10.569217356 +0000 UTC m=+2815.331654261"
Nov 02 08:11:11.241334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:11.240629    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:11.244103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:11.244058    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:11.244355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:11.244143    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:11.342334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:11.341642    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:11:11.342334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:11.341916    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy" containerID="cri-o://85764a84f101f826af9d202c8eeaaa50bda85759b0ee00a8ac6397c7d814a887" gracePeriod=30
Nov 02 08:11:12.240336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:12.235441    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:12.240336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:12.235522    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:13.235333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:13.234801    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:13.235333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:13.234866    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:14.236331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:14.235734    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:14.236331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:14.235799    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:15.236803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:15.236764    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:15.237479 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:15.237444    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:16.235003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.234959    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:16.235227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.235032    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:16.613668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.613630    4515 generic.go:356] "Generic (PLEG): container finished" podID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerID="85764a84f101f826af9d202c8eeaaa50bda85759b0ee00a8ac6397c7d814a887" exitCode=0
Nov 02 08:11:16.614098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.613700    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" event={"ID":"25b3f117-dd01-41d7-bf5f-66b45b3af982","Type":"ContainerDied","Data":"85764a84f101f826af9d202c8eeaaa50bda85759b0ee00a8ac6397c7d814a887"}
Nov 02 08:11:16.657569 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.657541    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:11:16.789886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789787    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.789886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789836    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.789886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789859    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.789886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789883    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789925    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.789983    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790013    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790073    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790136    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\") pod \"25b3f117-dd01-41d7-bf5f-66b45b3af982\" (UID: \"25b3f117-dd01-41d7-bf5f-66b45b3af982\") "
Nov 02 08:11:16.790262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790147    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:11:16.790635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790282    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:11:16.790635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790548    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:11:16.790794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790755    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:11:16.790929 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790902    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.790997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790936    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.790997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790952    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.790997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790967    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.790997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.790983    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:11:16.792515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.792486    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:11:16.792978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.792948    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:11:16.793105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.792977    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "kube-api-access-n4956". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:11:16.793630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.793606    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token" pod "25b3f117-dd01-41d7-bf5f-66b45b3af982" (UID: "25b3f117-dd01-41d7-bf5f-66b45b3af982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:11:16.892797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.892760    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b3f117-dd01-41d7-bf5f-66b45b3af982-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.892964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.892802    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.892964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.892819    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.892964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.892833    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:16.892964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:16.892847    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4956\" (UniqueName: \"kubernetes.io/projected/25b3f117-dd01-41d7-bf5f-66b45b3af982-kube-api-access-n4956\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:11:17.235389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.235275    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8qdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 02 08:11:17.235389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.235367    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 02 08:11:17.623972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.623935    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"
Nov 02 08:11:17.623972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.623968    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5" event={"ID":"25b3f117-dd01-41d7-bf5f-66b45b3af982","Type":"ContainerDied","Data":"852deccd11d58c1e6ccf0d45916013f0bb76ef70e610a3c8edc0a21396e40049"}
Nov 02 08:11:17.624652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.624035    4515 scope.go:122] "RemoveContainer" containerID="85764a84f101f826af9d202c8eeaaa50bda85759b0ee00a8ac6397c7d814a887"
Nov 02 08:11:17.691612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.691576    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:11:17.706879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:17.706839    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-gmpb5"]
Nov 02 08:11:17.727132 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:11:17.727092    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:11:18.346671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:18.346640    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:18.346995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:18.346972    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:18.348406 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:18.348381    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:11:19.087364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:19.087326    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:11:19.375589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:19.375514    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" path="/var/lib/kubelet/pods/25b3f117-dd01-41d7-bf5f-66b45b3af982/volumes"
Nov 02 08:11:44.807560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.807479    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:11:44.808233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.808217    4515 cpu_manager.go:414] "containerMap: removing container" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy"
Nov 02 08:11:44.808283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.808234    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy"
Nov 02 08:11:44.808368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.808358    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b3f117-dd01-41d7-bf5f-66b45b3af982" containerName="istio-proxy"
Nov 02 08:11:44.813235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.813211    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:44.830791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.830769    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"agc-cred-772dedcb61d7\""
Nov 02 08:11:44.843529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.843499    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"payload-processing-external-model-e2e--d97430f5-dockercfg-4kjcc\""
Nov 02 08:11:44.843980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.843706    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2365fa\""
Nov 02 08:11:44.843980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.843829    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"routing-overlay\""
Nov 02 08:11:44.843980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.843592    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"openshift-service-ca.crt\""
Nov 02 08:11:44.843980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.843841    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"kube-root-ca.crt\""
Nov 02 08:11:44.849184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.849154    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:11:44.927768 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.927724    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:44.927981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.927829    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:44.927981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.927869    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxjj\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:44.927981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:44.927951    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.035059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.034544    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.035059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.034621    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.035059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.034662    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxjj\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.035059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.034725    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.039328 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.036129    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.039328 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.036720    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.070371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.065655    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.070371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.066921    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxjj\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.138416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.138373    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.466667 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:11:45.466636    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e65a5e_b150_431c_93af_7cf25d19cd4e.slice/crio-b07ea0422e161effaeb45d5598baa239748d2960e66e3b8807d21c005427bf1a WatchSource:0}: Error finding container b07ea0422e161effaeb45d5598baa239748d2960e66e3b8807d21c005427bf1a: Status 404 returned error can't find the container with id b07ea0422e161effaeb45d5598baa239748d2960e66e3b8807d21c005427bf1a
Nov 02 08:11:45.468143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.468117    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:11:45.868185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.868143    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" event={"ID":"d2e65a5e-b150-431c-93af-7cf25d19cd4e","Type":"ContainerStarted","Data":"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"}
Nov 02 08:11:45.868638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.868213    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:11:45.868638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.868234    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" event={"ID":"d2e65a5e-b150-431c-93af-7cf25d19cd4e","Type":"ContainerStarted","Data":"b07ea0422e161effaeb45d5598baa239748d2960e66e3b8807d21c005427bf1a"}
Nov 02 08:11:45.912611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:45.912557    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" podStartSLOduration=1.9125416309999999 podStartE2EDuration="1.912541631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:11:45.909414297 +0000 UTC m=+2850.671851202" watchObservedRunningTime="2026-11-02 08:11:45.912541631 +0000 UTC m=+2850.674978537"
Nov 02 08:11:47.724037 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:11:47.724000    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:11:56.878519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:11:56.878488    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:12:09.191844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.191809    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:09.196760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.196741    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.216067 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.216035    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:09.310288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.310250    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.310513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.310449    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.310513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.310508    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qlh\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.310634 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.310532    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.412212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.412176    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.412438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.412247    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qlh\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.412438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.412276    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.412575 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.412498    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.413107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.413071    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.413227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.413208    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.414668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.414648    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.486390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.484096    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qlh\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\") pod \"payload-processing-external-model-58ddf7f69-plx99\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") " pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.521263 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.519937    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:09.879264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.879230    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:12:09.939402 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:12:09.939371    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f91c50_8fd1_4dc9_98a3_7edc930778b6.slice/crio-9f090271871befaf49ba1cfa2ebf29f4562dc8c78d8061400798f87537d99bd7 WatchSource:0}: Error finding container 9f090271871befaf49ba1cfa2ebf29f4562dc8c78d8061400798f87537d99bd7: Status 404 returned error can't find the container with id 9f090271871befaf49ba1cfa2ebf29f4562dc8c78d8061400798f87537d99bd7
Nov 02 08:12:09.942867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:09.942841    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:10.131425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.131343    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" event={"ID":"92f91c50-8fd1-4dc9-98a3-7edc930778b6","Type":"ContainerStarted","Data":"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"}
Nov 02 08:12:10.131425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.131403    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" event={"ID":"92f91c50-8fd1-4dc9-98a3-7edc930778b6","Type":"ContainerStarted","Data":"9f090271871befaf49ba1cfa2ebf29f4562dc8c78d8061400798f87537d99bd7"}
Nov 02 08:12:10.131641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.131538    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:10.178586 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.178524    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" podStartSLOduration=1.178505183 podStartE2EDuration="1.178505183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:10.172698657 +0000 UTC m=+2874.935135586" watchObservedRunningTime="2026-11-02 08:12:10.178505183 +0000 UTC m=+2874.940942092"
Nov 02 08:12:10.293334 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.283702    4515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-621795298e14: secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.293334 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.283735    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58ddf7f69-plx99: secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.293334 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.283797    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:10.783777075 +0000 UTC m=+2875.546213978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.791987 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.791951    4515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-621795298e14: secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.791987 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.791982    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58ddf7f69-plx99: secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.792216 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:10.792065    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:11.792045961 +0000 UTC m=+2876.554482871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : secret "agc-cred-621795298e14" not found
Nov 02 08:12:10.819438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.819405    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:12:10.837706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:10.837675    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:11.139015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:11.138933    4515 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" secret="" err="secret \"payload-processing-external-model-dockercfg-l2lh7\" not found"
Nov 02 08:12:11.309335 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.309112    4515 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:11.309335 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.309192    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:11.809173145 +0000 UTC m=+2876.571610043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:11.838867 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.838831    4515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-621795298e14: secret "agc-cred-621795298e14" not found
Nov 02 08:12:11.838867 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.838862    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58ddf7f69-plx99: secret "agc-cred-621795298e14" not found
Nov 02 08:12:11.839131 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.838919    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:13.838904306 +0000 UTC m=+2878.601341216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : secret "agc-cred-621795298e14" not found
Nov 02 08:12:11.839131 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.838948    4515 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:11.839131 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:11.839018    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:12.839000752 +0000 UTC m=+2877.601437649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:12.859531 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:12.859497    4515 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:12.859965 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:12.859573    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:14.859554681 +0000 UTC m=+2879.621991570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:13.441157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.441113    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:12:13.445892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.445860    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.465748 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.465719    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:12:13.467921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.467890    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.468068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.468053    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.468218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.468203    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8c9\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.468288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.468236    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.472348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472321    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-52831c4c288c\""
Nov 02 08:12:13.472497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472361    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:12:13.472497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472420    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-e2e--c1e7f36e-dockercfg-52r9d\""
Nov 02 08:12:13.472497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472438    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"kube-root-ca.crt\""
Nov 02 08:12:13.472785 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472763    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"routing-overlay\""
Nov 02 08:12:13.472915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.472866    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"openshift-service-ca.crt\""
Nov 02 08:12:13.600331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.578392    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8c9\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.604337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.604285    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.632466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.632429    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.632854 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.632829    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.634072 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.634044    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.668332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.654031    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.679369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.672155    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.684334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.682083    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8c9\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.763886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:13.763847    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:13.869635 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:13.869601    4515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-621795298e14: secret "agc-cred-621795298e14" not found
Nov 02 08:12:13.870162 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:13.870144    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58ddf7f69-plx99: secret "agc-cred-621795298e14" not found
Nov 02 08:12:13.870399 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:13.870377    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:17.870354544 +0000 UTC m=+2882.632791453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : secret "agc-cred-621795298e14" not found
Nov 02 08:12:14.268731 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:12:14.268669    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5d7b35_81e4_45a9_b37d_fc45ba8e5b98.slice/crio-40a8d505a1c4b03842238f77fcf6d3153ec12d146dff8079d3e7f4bbec248419 WatchSource:0}: Error finding container 40a8d505a1c4b03842238f77fcf6d3153ec12d146dff8079d3e7f4bbec248419: Status 404 returned error can't find the container with id 40a8d505a1c4b03842238f77fcf6d3153ec12d146dff8079d3e7f4bbec248419
Nov 02 08:12:14.270836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:14.270804    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:12:14.913156 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:14.913117    4515 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:14.913770 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:14.913219    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume podName:92f91c50-8fd1-4dc9-98a3-7edc930778b6 nodeName:}" failed. No retries permitted until 2026-11-02 08:12:18.913198334 +0000 UTC m=+2883.675635220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume") pod "payload-processing-external-model-58ddf7f69-plx99" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6") : configmap "payload-processing-external-model-plugins" not found
Nov 02 08:12:15.171184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:15.171096    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" event={"ID":"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98","Type":"ContainerStarted","Data":"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"}
Nov 02 08:12:15.171184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:15.171161    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" event={"ID":"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98","Type":"ContainerStarted","Data":"40a8d505a1c4b03842238f77fcf6d3153ec12d146dff8079d3e7f4bbec248419"}
Nov 02 08:12:15.171400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:15.171191    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:15.216687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:15.216632    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" podStartSLOduration=2.216616932 podStartE2EDuration="2.216616932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:15.212481299 +0000 UTC m=+2879.974918226" watchObservedRunningTime="2026-11-02 08:12:15.216616932 +0000 UTC m=+2879.979053837"
Nov 02 08:12:15.820619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:15.820574    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" podUID="e7c654f3-32e4-4a94-88ff-186098c11311" containerName="payload-processing" containerID="cri-o://5824a98f7d3052fd2ce73bfc0b2109c2f970224b1569fdbbf5c093faa266709f" gracePeriod=25
Nov 02 08:12:16.188107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.188074    4515 generic.go:356] "Generic (PLEG): container finished" podID="e7c654f3-32e4-4a94-88ff-186098c11311" containerID="5824a98f7d3052fd2ce73bfc0b2109c2f970224b1569fdbbf5c093faa266709f" exitCode=0
Nov 02 08:12:16.189692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.189664    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" event={"ID":"e7c654f3-32e4-4a94-88ff-186098c11311","Type":"ContainerDied","Data":"5824a98f7d3052fd2ce73bfc0b2109c2f970224b1569fdbbf5c093faa266709f"}
Nov 02 08:12:16.277510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.277484    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:12:16.352986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.352944    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\") pod \"e7c654f3-32e4-4a94-88ff-186098c11311\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") "
Nov 02 08:12:16.353222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.353064    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\") pod \"e7c654f3-32e4-4a94-88ff-186098c11311\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") "
Nov 02 08:12:16.353222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.353191    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\") pod \"e7c654f3-32e4-4a94-88ff-186098c11311\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") "
Nov 02 08:12:16.353396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.353276    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\") pod \"e7c654f3-32e4-4a94-88ff-186098c11311\" (UID: \"e7c654f3-32e4-4a94-88ff-186098c11311\") "
Nov 02 08:12:16.354579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.354527    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume" pod "e7c654f3-32e4-4a94-88ff-186098c11311" (UID: "e7c654f3-32e4-4a94-88ff-186098c11311"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:16.354728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.354683    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:16.355203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.355154    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume" pod "e7c654f3-32e4-4a94-88ff-186098c11311" (UID: "e7c654f3-32e4-4a94-88ff-186098c11311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:16.356713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.356682    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials" pod "e7c654f3-32e4-4a94-88ff-186098c11311" (UID: "e7c654f3-32e4-4a94-88ff-186098c11311"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:16.356916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.356896    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89" pod "e7c654f3-32e4-4a94-88ff-186098c11311" (UID: "e7c654f3-32e4-4a94-88ff-186098c11311"). InnerVolumeSpecName "kube-api-access-64j89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:16.456513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.456428    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:16.456513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.456463    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c654f3-32e4-4a94-88ff-186098c11311-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:16.456513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:16.456481    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j89\" (UniqueName: \"kubernetes.io/projected/e7c654f3-32e4-4a94-88ff-186098c11311-kube-api-access-64j89\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:17.147490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.147446    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" podUID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" containerName="payload-processing" containerID="cri-o://969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3" gracePeriod=25
Nov 02 08:12:17.198180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.198151    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w"
Nov 02 08:12:17.198679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.198185    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-744487595b-p774w" event={"ID":"e7c654f3-32e4-4a94-88ff-186098c11311","Type":"ContainerDied","Data":"03cab93447a8d2325630c2369a1a34bb1769cde3a0335cbbcf84f5e8a59ef821"}
Nov 02 08:12:17.198679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.198244    4515 scope.go:122] "RemoveContainer" containerID="5824a98f7d3052fd2ce73bfc0b2109c2f970224b1569fdbbf5c093faa266709f"
Nov 02 08:12:17.264468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.264437    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:12:17.286440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.286400    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-744487595b-p774w"]
Nov 02 08:12:17.379413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.379382    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c654f3-32e4-4a94-88ff-186098c11311" path="/var/lib/kubelet/pods/e7c654f3-32e4-4a94-88ff-186098c11311/volumes"
Nov 02 08:12:17.441453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.441425    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:17.471832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.471804    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\") pod \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") "
Nov 02 08:12:17.472018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.471847    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\") pod \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") "
Nov 02 08:12:17.472018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.471924    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\") pod \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") "
Nov 02 08:12:17.472018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.472009    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\") pod \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\" (UID: \"92f91c50-8fd1-4dc9-98a3-7edc930778b6\") "
Nov 02 08:12:17.472457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.472432    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume" pod "92f91c50-8fd1-4dc9-98a3-7edc930778b6" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:17.472631 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.472457    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume" pod "92f91c50-8fd1-4dc9-98a3-7edc930778b6" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:17.472847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.472828    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:17.472847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.472847    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f91c50-8fd1-4dc9-98a3-7edc930778b6-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:17.474147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.474127    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh" pod "92f91c50-8fd1-4dc9-98a3-7edc930778b6" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6"). InnerVolumeSpecName "kube-api-access-s8qlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:17.474251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.474165    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials" pod "92f91c50-8fd1-4dc9-98a3-7edc930778b6" (UID: "92f91c50-8fd1-4dc9-98a3-7edc930778b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:17.574590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.574552    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8qlh\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-kube-api-access-s8qlh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:17.574590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:17.574585    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f91c50-8fd1-4dc9-98a3-7edc930778b6-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:17.725057 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:17.724974    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:12:18.207350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.207285    4515 generic.go:356] "Generic (PLEG): container finished" podID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" containerID="969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3" exitCode=0
Nov 02 08:12:18.207842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.207368    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"
Nov 02 08:12:18.207842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.207368    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" event={"ID":"92f91c50-8fd1-4dc9-98a3-7edc930778b6","Type":"ContainerDied","Data":"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"}
Nov 02 08:12:18.207842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.207486    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58ddf7f69-plx99" event={"ID":"92f91c50-8fd1-4dc9-98a3-7edc930778b6","Type":"ContainerDied","Data":"9f090271871befaf49ba1cfa2ebf29f4562dc8c78d8061400798f87537d99bd7"}
Nov 02 08:12:18.207842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.207514    4515 scope.go:122] "RemoveContainer" containerID="969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"
Nov 02 08:12:18.218096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.218078    4515 scope.go:122] "RemoveContainer" containerID="969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"
Nov 02 08:12:18.218378 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:18.218357    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3\": container with ID starting with 969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3 not found: ID does not exist" containerID="969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"
Nov 02 08:12:18.218453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.218386    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3"} err="failed to get container status \"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3\": rpc error: code = NotFound desc = could not find container \"969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3\": container with ID starting with 969f22b91f2fdef1304c44f97404f95ee20537f2cb332214b4b7c06e3a6774b3 not found: ID does not exist"
Nov 02 08:12:18.276843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.276803    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:18.289740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:18.289704    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58ddf7f69-plx99"]
Nov 02 08:12:19.391997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:19.391957    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" path="/var/lib/kubelet/pods/92f91c50-8fd1-4dc9-98a3-7edc930778b6/volumes"
Nov 02 08:12:19.943509 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:19.943477    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 08:12:19.943697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:19.943666    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" containerID="cri-o://badf15720331329e6ce7332c3729435bac87ac53dbd46e48f829fe6fdd9d386e" gracePeriod=30
Nov 02 08:12:21.244052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:21.244018    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:12:21.244464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:21.244076    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:12:25.311249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.311208    4515 generic.go:356] "Generic (PLEG): container finished" podID="98d605b9-1283-437c-b85e-e7bc74264388" containerID="badf15720331329e6ce7332c3729435bac87ac53dbd46e48f829fe6fdd9d386e" exitCode=0
Nov 02 08:12:25.311249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.311238    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" event={"ID":"98d605b9-1283-437c-b85e-e7bc74264388","Type":"ContainerDied","Data":"badf15720331329e6ce7332c3729435bac87ac53dbd46e48f829fe6fdd9d386e"}
Nov 02 08:12:25.351570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.351541    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 08:12:25.520920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.520885    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.520963    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521001    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521057    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521230    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:12:25.521656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521250    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521367    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521503    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521604    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\") pod \"98d605b9-1283-437c-b85e-e7bc74264388\" (UID: \"98d605b9-1283-437c-b85e-e7bc74264388\") "
Nov 02 08:12:25.521937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.521818    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:25.522322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.522008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:12:25.522425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.522406    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.522488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.522427    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.522488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.522443    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d605b9-1283-437c-b85e-e7bc74264388-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.523090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.523053    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:12:25.523939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.523601    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:12:25.524476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.524389    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "kube-api-access-7bd8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:25.524594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.524504    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:12:25.524870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.524844    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:12:25.525766 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.525740    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token" pod "98d605b9-1283-437c-b85e-e7bc74264388" (UID: "98d605b9-1283-437c-b85e-e7bc74264388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:25.618974 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.618891    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:25.619677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619658    4515 cpu_manager.go:414] "containerMap: removing container" podUID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" containerName="payload-processing"
Nov 02 08:12:25.619677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619677    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" containerName="payload-processing"
Nov 02 08:12:25.619812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619717    4515 cpu_manager.go:414] "containerMap: removing container" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy"
Nov 02 08:12:25.619812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619723    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy"
Nov 02 08:12:25.619812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619742    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e7c654f3-32e4-4a94-88ff-186098c11311" containerName="payload-processing"
Nov 02 08:12:25.619812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619750    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c654f3-32e4-4a94-88ff-186098c11311" containerName="payload-processing"
Nov 02 08:12:25.619969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619865    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c654f3-32e4-4a94-88ff-186098c11311" containerName="payload-processing"
Nov 02 08:12:25.619969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619879    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f91c50-8fd1-4dc9-98a3-7edc930778b6" containerName="payload-processing"
Nov 02 08:12:25.619969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.619892    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d605b9-1283-437c-b85e-e7bc74264388" containerName="istio-proxy"
Nov 02 08:12:25.623790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.623774    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.624401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624375    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.624485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624407    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.624485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624423    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bd8d\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-kube-api-access-7bd8d\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.624485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624439    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d605b9-1283-437c-b85e-e7bc74264388-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.624485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624454    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d605b9-1283-437c-b85e-e7bc74264388-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.624485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.624467    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d605b9-1283-437c-b85e-e7bc74264388-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641573    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641658    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n2sww\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641670    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641742    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641763    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 08:12:25.641830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.641804    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c68363570711\""
Nov 02 08:12:25.644107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.644081    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:25.727796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.727742    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.728172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.728142    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5872f\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.729537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.729115    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.729537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.729206    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.835341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.832484    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.835341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.832578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.835341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.832707    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.835341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.832778    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5872f\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.861557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.839281    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.861557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.839919    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.861557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.857852    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.872387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.870082    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5872f\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\") pod \"payload-processing-external-model-5bc4454848-fs5kr\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:25.941800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:25.941211    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:26.193300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.193215    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:12:26.281766 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.281741    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:26.321665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.321622    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" event={"ID":"c6a5743c-a2b0-4101-9a05-52b326b15b18","Type":"ContainerStarted","Data":"9c8cabad8a244a6ba7df72f1111cbeeac77afb962b26fd76da177e9cc2491822"}
Nov 02 08:12:26.323208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.323179    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc" event={"ID":"98d605b9-1283-437c-b85e-e7bc74264388","Type":"ContainerDied","Data":"1a2842cf0c3d4ab0ad3a935483b126d2d35d1641b6ab6469e83168b72d906fb8"}
Nov 02 08:12:26.323208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.323199    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"
Nov 02 08:12:26.323419 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.323238    4515 scope.go:122] "RemoveContainer" containerID="badf15720331329e6ce7332c3729435bac87ac53dbd46e48f829fe6fdd9d386e"
Nov 02 08:12:26.411147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.411116    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 08:12:26.450505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:26.450425    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-r5cvc"]
Nov 02 08:12:27.332519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:27.332482    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" event={"ID":"c6a5743c-a2b0-4101-9a05-52b326b15b18","Type":"ContainerStarted","Data":"3f4713687d95e3acfc7256abcf5679817b0ccf2225530a9df63c6ee881995645"}
Nov 02 08:12:27.333016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:27.332572    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:27.375216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:27.375176    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d605b9-1283-437c-b85e-e7bc74264388" path="/var/lib/kubelet/pods/98d605b9-1283-437c-b85e-e7bc74264388/volumes"
Nov 02 08:12:27.377520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:27.377475    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" podStartSLOduration=2.377460764 podStartE2EDuration="2.377460764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:27.373814161 +0000 UTC m=+2892.136251067" watchObservedRunningTime="2026-11-02 08:12:27.377460764 +0000 UTC m=+2892.139897712"
Nov 02 08:12:31.132279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:31.132240    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:31.136163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:31.136133    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:35.798354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.790994    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:12:35.808350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.808324    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.813662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.813631    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:12:35.873039 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.872991    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.875407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.875381    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.875723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.875703    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tts\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.875875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.875857    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.876728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.876705    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:12:35.876968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.876948    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:12:35.880901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.880881    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-9af54fb2dfa8\""
Nov 02 08:12:35.883931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.881251    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:12:35.884094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.881291    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:12:35.884230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.881987    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-vppdr\""
Nov 02 08:12:35.977441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.977397    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tts\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.978138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.978111    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.978967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.978940    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.979687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.979663    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.980654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.979335    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.980875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.980479    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:35.984980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:35.984961    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:36.065185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.065095    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tts\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:36.138436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.138384    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" podUID="c6a5743c-a2b0-4101-9a05-52b326b15b18" containerName="payload-processing" containerID="cri-o://3f4713687d95e3acfc7256abcf5679817b0ccf2225530a9df63c6ee881995645" gracePeriod=25
Nov 02 08:12:36.212623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.212515    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:36.707384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.706886    4515 generic.go:356] "Generic (PLEG): container finished" podID="c6a5743c-a2b0-4101-9a05-52b326b15b18" containerID="3f4713687d95e3acfc7256abcf5679817b0ccf2225530a9df63c6ee881995645" exitCode=0
Nov 02 08:12:36.707384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.707039    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" event={"ID":"c6a5743c-a2b0-4101-9a05-52b326b15b18","Type":"ContainerDied","Data":"3f4713687d95e3acfc7256abcf5679817b0ccf2225530a9df63c6ee881995645"}
Nov 02 08:12:36.736333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:36.715642    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:12:36.773926 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:12:36.773886    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d15ebc_b368_458e_a41a_f5b354fd98ef.slice/crio-bd53804b40bc0d03a7a61a8e2f603e54100d112125529e656774fd46bde357ae WatchSource:0}: Error finding container bd53804b40bc0d03a7a61a8e2f603e54100d112125529e656774fd46bde357ae: Status 404 returned error can't find the container with id bd53804b40bc0d03a7a61a8e2f603e54100d112125529e656774fd46bde357ae
Nov 02 08:12:37.130570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.119224    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:12:37.162368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.162225    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:12:37.162546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.162440    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.186329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.184037    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"kube-root-ca.crt\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.195837    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"routing-overlay\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.195963    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"payload-processing-external-model-e2e--4bba7e07-dockercfg-gp4vp\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.196014    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"agc-cred-507d8cea5cf2\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.196080    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696c7a\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.196160    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"openshift-service-ca.crt\""
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.199244    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.199444    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.199502    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8978f\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.199743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.199576    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.307023    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.307156    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.307206    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8978f\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.307270    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.308170    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.319739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.310862    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.364333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.362873    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8978f\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.437341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.434556    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:37.475708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.475676    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.546435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.533063    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\") pod \"c6a5743c-a2b0-4101-9a05-52b326b15b18\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") "
Nov 02 08:12:37.546435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.533386    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\") pod \"c6a5743c-a2b0-4101-9a05-52b326b15b18\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") "
Nov 02 08:12:37.546435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.533459    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\") pod \"c6a5743c-a2b0-4101-9a05-52b326b15b18\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") "
Nov 02 08:12:37.546435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.533501    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\") pod \"c6a5743c-a2b0-4101-9a05-52b326b15b18\" (UID: \"c6a5743c-a2b0-4101-9a05-52b326b15b18\") "
Nov 02 08:12:37.546435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.541003    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume" pod "c6a5743c-a2b0-4101-9a05-52b326b15b18" (UID: "c6a5743c-a2b0-4101-9a05-52b326b15b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:37.547197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.547079    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume" pod "c6a5743c-a2b0-4101-9a05-52b326b15b18" (UID: "c6a5743c-a2b0-4101-9a05-52b326b15b18"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:37.572966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.572687    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:37.601550 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.585761    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials" pod "c6a5743c-a2b0-4101-9a05-52b326b15b18" (UID: "c6a5743c-a2b0-4101-9a05-52b326b15b18"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:37.603365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.601985    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f" pod "c6a5743c-a2b0-4101-9a05-52b326b15b18" (UID: "c6a5743c-a2b0-4101-9a05-52b326b15b18"). InnerVolumeSpecName "kube-api-access-5872f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:37.656607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.636285    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5872f\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-kube-api-access-5872f\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:37.656607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.636350    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:37.656607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.636370    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6a5743c-a2b0-4101-9a05-52b326b15b18-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:37.656607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.636385    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6a5743c-a2b0-4101-9a05-52b326b15b18-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:12:37.800381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.799727    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"
Nov 02 08:12:37.801385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.799770    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr" event={"ID":"c6a5743c-a2b0-4101-9a05-52b326b15b18","Type":"ContainerDied","Data":"9c8cabad8a244a6ba7df72f1111cbeeac77afb962b26fd76da177e9cc2491822"}
Nov 02 08:12:37.801521 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.801431    4515 scope.go:122] "RemoveContainer" containerID="3f4713687d95e3acfc7256abcf5679817b0ccf2225530a9df63c6ee881995645"
Nov 02 08:12:37.802743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.802619    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" event={"ID":"21d15ebc-b368-458e-a41a-f5b354fd98ef","Type":"ContainerStarted","Data":"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"}
Nov 02 08:12:37.802743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.802660    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" event={"ID":"21d15ebc-b368-458e-a41a-f5b354fd98ef","Type":"ContainerStarted","Data":"bd53804b40bc0d03a7a61a8e2f603e54100d112125529e656774fd46bde357ae"}
Nov 02 08:12:37.802880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.802773    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:37.909643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.909611    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:37.915493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.914658    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-fs5kr"]
Nov 02 08:12:37.981332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:37.980774    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" podStartSLOduration=2.98075044 podStartE2EDuration="2.98075044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:37.975667076 +0000 UTC m=+2902.738103983" watchObservedRunningTime="2026-11-02 08:12:37.98075044 +0000 UTC m=+2902.743187346"
Nov 02 08:12:38.395207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:38.395170    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:12:38.813917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:38.813880    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" event={"ID":"3726e13e-8216-4a83-a4bf-4e7ad4ca954f","Type":"ContainerStarted","Data":"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"}
Nov 02 08:12:38.814234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:38.814213    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:38.814414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:38.814396    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" event={"ID":"3726e13e-8216-4a83-a4bf-4e7ad4ca954f","Type":"ContainerStarted","Data":"44609128e4a50ca901eee5593c8586ffd0665c54e47d5c8302619dc43f04e111"}
Nov 02 08:12:38.862630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:38.862567    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" podStartSLOduration=1.862547905 podStartE2EDuration="1.862547905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:38.862347216 +0000 UTC m=+2903.624784124" watchObservedRunningTime="2026-11-02 08:12:38.862547905 +0000 UTC m=+2903.624984814"
Nov 02 08:12:39.376103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:39.376068    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a5743c-a2b0-4101-9a05-52b326b15b18" path="/var/lib/kubelet/pods/c6a5743c-a2b0-4101-9a05-52b326b15b18/volumes"
Nov 02 08:12:47.726726 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:12:47.726687    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:12:48.819113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:48.819084    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:12:49.838831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:49.838802    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:12:53.267974 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.267941    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:12:53.268860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.268838    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c6a5743c-a2b0-4101-9a05-52b326b15b18" containerName="payload-processing"
Nov 02 08:12:53.268860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.268862    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a5743c-a2b0-4101-9a05-52b326b15b18" containerName="payload-processing"
Nov 02 08:12:53.268990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.268968    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a5743c-a2b0-4101-9a05-52b326b15b18" containerName="payload-processing"
Nov 02 08:12:53.273643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.273622    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.301200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.301169    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-3864b59b624b\""
Nov 02 08:12:53.305078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.305046    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:12:53.382218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.382179    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzrx\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.382404 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.382339    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.382404 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.382400    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.382514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.382498    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.483649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.483616    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.483845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.483694    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.483845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.483817    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.484008 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.483988    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzrx\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.484506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.484482    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.484638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.484614    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.486405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.486382    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.507946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.507917    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzrx\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\") pod \"payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.594380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.592968    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:53.946790 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:12:53.946754    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ac439e_5a3f_48da_8ca7_98faa0b4acd8.slice/crio-3b15c1c2b2e11fda913c3eee6303cd66c6d438ebc89f4cbebd8689213741235c WatchSource:0}: Error finding container 3b15c1c2b2e11fda913c3eee6303cd66c6d438ebc89f4cbebd8689213741235c: Status 404 returned error can't find the container with id 3b15c1c2b2e11fda913c3eee6303cd66c6d438ebc89f4cbebd8689213741235c
Nov 02 08:12:53.948882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:53.948848    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:12:54.024758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.024715    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" event={"ID":"46ac439e-5a3f-48da-8ca7-98faa0b4acd8","Type":"ContainerStarted","Data":"3b15c1c2b2e11fda913c3eee6303cd66c6d438ebc89f4cbebd8689213741235c"}
Nov 02 08:12:54.132198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.132157    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:12:54.136860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.136839    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.152554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.152527    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"agc-cred-0bc55f0936eb\""
Nov 02 08:12:54.157006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.156971    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:12:54.306897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.306858    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.306897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.306912    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrzlx\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.307501 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.307068    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.307501 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.307257    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.408299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.408264    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.408526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.408428    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.408526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.408459    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrzlx\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.408526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.408508    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.409081 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.409057    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.409213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.409176    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.411047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.411030    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.432287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.432260    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrzlx\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.452674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.452644    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:54.763982 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:12:54.763952    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25c00df_5c8b_488b_a7f7_06adaa66c16b.slice/crio-74919cdc3a860cbe6deda627cee01e2c80b2f605f9ed129f2b7f764a0c94bf8c WatchSource:0}: Error finding container 74919cdc3a860cbe6deda627cee01e2c80b2f605f9ed129f2b7f764a0c94bf8c: Status 404 returned error can't find the container with id 74919cdc3a860cbe6deda627cee01e2c80b2f605f9ed129f2b7f764a0c94bf8c
Nov 02 08:12:54.765832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:54.765804    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:12:55.037400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.034986    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" event={"ID":"f25c00df-5c8b-488b-a7f7-06adaa66c16b","Type":"ContainerStarted","Data":"feb9e1065a9203691268c92c31fcc815b75a5aa9608c4e386908220998171a48"}
Nov 02 08:12:55.037400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.035067    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" event={"ID":"f25c00df-5c8b-488b-a7f7-06adaa66c16b","Type":"ContainerStarted","Data":"74919cdc3a860cbe6deda627cee01e2c80b2f605f9ed129f2b7f764a0c94bf8c"}
Nov 02 08:12:55.037400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.035097    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:12:55.037400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.037028    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" event={"ID":"46ac439e-5a3f-48da-8ca7-98faa0b4acd8","Type":"ContainerStarted","Data":"769d9b5a58a2d0f63cca10361b6aed86f54ba6abe20447d243338bc0a633d236"}
Nov 02 08:12:55.037400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.037170    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:12:55.077602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.077549    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" podStartSLOduration=1.07753625 podStartE2EDuration="1.07753625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:55.076226248 +0000 UTC m=+2919.838663154" watchObservedRunningTime="2026-11-02 08:12:55.07753625 +0000 UTC m=+2919.839973156"
Nov 02 08:12:55.121592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:12:55.121539    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" podStartSLOduration=2.121525267 podStartE2EDuration="2.121525267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:55.118048965 +0000 UTC m=+2919.880485872" watchObservedRunningTime="2026-11-02 08:12:55.121525267 +0000 UTC m=+2919.883962172"
Nov 02 08:13:06.047543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:06.047461    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:13:06.048030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:06.047614    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:13:06.124116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:06.124084    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:13:06.168080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:06.168042    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:13:11.124706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.124663    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" podUID="21d15ebc-b368-458e-a41a-f5b354fd98ef" containerName="payload-processing" containerID="cri-o://8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663" gracePeriod=25
Nov 02 08:13:11.181374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.169927    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" podUID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" containerName="payload-processing" containerID="cri-o://f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358" gracePeriod=25
Nov 02 08:13:11.489775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.489739    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:13:11.493248 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.493224    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\") pod \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") "
Nov 02 08:13:11.493455 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.493441    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\") pod \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") "
Nov 02 08:13:11.493543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.493468    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\") pod \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") "
Nov 02 08:13:11.493543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.493529    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\") pod \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\" (UID: \"3726e13e-8216-4a83-a4bf-4e7ad4ca954f\") "
Nov 02 08:13:11.493700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.493674    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume" pod "3726e13e-8216-4a83-a4bf-4e7ad4ca954f" (UID: "3726e13e-8216-4a83-a4bf-4e7ad4ca954f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:11.494207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.494180    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.494340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.494211    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume" pod "3726e13e-8216-4a83-a4bf-4e7ad4ca954f" (UID: "3726e13e-8216-4a83-a4bf-4e7ad4ca954f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:11.496809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.496778    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f" pod "3726e13e-8216-4a83-a4bf-4e7ad4ca954f" (UID: "3726e13e-8216-4a83-a4bf-4e7ad4ca954f"). InnerVolumeSpecName "kube-api-access-8978f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:11.496914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.496811    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials" pod "3726e13e-8216-4a83-a4bf-4e7ad4ca954f" (UID: "3726e13e-8216-4a83-a4bf-4e7ad4ca954f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:11.531817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.531787    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:13:11.595080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.595045    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\") pod \"21d15ebc-b368-458e-a41a-f5b354fd98ef\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") "
Nov 02 08:13:11.595424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.595114    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\") pod \"21d15ebc-b368-458e-a41a-f5b354fd98ef\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") "
Nov 02 08:13:11.595424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.595191    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\") pod \"21d15ebc-b368-458e-a41a-f5b354fd98ef\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") "
Nov 02 08:13:11.595424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.595289    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\") pod \"21d15ebc-b368-458e-a41a-f5b354fd98ef\" (UID: \"21d15ebc-b368-458e-a41a-f5b354fd98ef\") "
Nov 02 08:13:11.595724 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.595688    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume" pod "21d15ebc-b368-458e-a41a-f5b354fd98ef" (UID: "21d15ebc-b368-458e-a41a-f5b354fd98ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:11.596026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.596001    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.596108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.596033    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.596108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.596048    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8978f\" (UniqueName: \"kubernetes.io/projected/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-kube-api-access-8978f\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.596108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.596062    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3726e13e-8216-4a83-a4bf-4e7ad4ca954f-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.598268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.598220    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials" pod "21d15ebc-b368-458e-a41a-f5b354fd98ef" (UID: "21d15ebc-b368-458e-a41a-f5b354fd98ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:11.598268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.598257    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts" pod "21d15ebc-b368-458e-a41a-f5b354fd98ef" (UID: "21d15ebc-b368-458e-a41a-f5b354fd98ef"). InnerVolumeSpecName "kube-api-access-64tts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:11.602652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.602624    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume" pod "21d15ebc-b368-458e-a41a-f5b354fd98ef" (UID: "21d15ebc-b368-458e-a41a-f5b354fd98ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:11.697429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.697345    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.697429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.697382    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tts\" (UniqueName: \"kubernetes.io/projected/21d15ebc-b368-458e-a41a-f5b354fd98ef-kube-api-access-64tts\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:11.697429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:11.697392    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21d15ebc-b368-458e-a41a-f5b354fd98ef-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:12.214068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.214036    4515 generic.go:356] "Generic (PLEG): container finished" podID="21d15ebc-b368-458e-a41a-f5b354fd98ef" containerID="8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663" exitCode=0
Nov 02 08:13:12.214562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.214084    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"
Nov 02 08:13:12.214562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.214122    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" event={"ID":"21d15ebc-b368-458e-a41a-f5b354fd98ef","Type":"ContainerDied","Data":"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"}
Nov 02 08:13:12.214562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.214180    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s" event={"ID":"21d15ebc-b368-458e-a41a-f5b354fd98ef","Type":"ContainerDied","Data":"bd53804b40bc0d03a7a61a8e2f603e54100d112125529e656774fd46bde357ae"}
Nov 02 08:13:12.214562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.214213    4515 scope.go:122] "RemoveContainer" containerID="8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"
Nov 02 08:13:12.215643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.215609    4515 generic.go:356] "Generic (PLEG): container finished" podID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" containerID="f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358" exitCode=0
Nov 02 08:13:12.215702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.215683    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" event={"ID":"3726e13e-8216-4a83-a4bf-4e7ad4ca954f","Type":"ContainerDied","Data":"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"}
Nov 02 08:13:12.215749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.215713    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g" event={"ID":"3726e13e-8216-4a83-a4bf-4e7ad4ca954f","Type":"ContainerDied","Data":"44609128e4a50ca901eee5593c8586ffd0665c54e47d5c8302619dc43f04e111"}
Nov 02 08:13:12.215787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.215771    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"
Nov 02 08:13:12.226127 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.226052    4515 scope.go:122] "RemoveContainer" containerID="8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"
Nov 02 08:13:12.226434 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:12.226411    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663\": container with ID starting with 8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663 not found: ID does not exist" containerID="8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"
Nov 02 08:13:12.226508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.226441    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663"} err="failed to get container status \"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663\": rpc error: code = NotFound desc = could not find container \"8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663\": container with ID starting with 8c0a3e56a84a1b24b0d4cce8317841a455ea150e5652a47a722d959d63fd5663 not found: ID does not exist"
Nov 02 08:13:12.226508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.226461    4515 scope.go:122] "RemoveContainer" containerID="f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"
Nov 02 08:13:12.235678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.235657    4515 scope.go:122] "RemoveContainer" containerID="f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"
Nov 02 08:13:12.235964 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:12.235943    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358\": container with ID starting with f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358 not found: ID does not exist" containerID="f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"
Nov 02 08:13:12.236013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.235970    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358"} err="failed to get container status \"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358\": rpc error: code = NotFound desc = could not find container \"f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358\": container with ID starting with f107a8c5fa61c502ac5e37f9819bad2532e66a96ab69cc3c6329205ed28e0358 not found: ID does not exist"
Nov 02 08:13:12.268894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.268862    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:13:12.281227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.281198    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4lwg5s"]
Nov 02 08:13:12.309192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.309152    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:13:12.313515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:12.313489    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c772fr7g"]
Nov 02 08:13:13.189972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.189940    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:13.190752 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190736    4515 cpu_manager.go:414] "containerMap: removing container" podUID="21d15ebc-b368-458e-a41a-f5b354fd98ef" containerName="payload-processing"
Nov 02 08:13:13.190810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190754    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d15ebc-b368-458e-a41a-f5b354fd98ef" containerName="payload-processing"
Nov 02 08:13:13.190810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190783    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" containerName="payload-processing"
Nov 02 08:13:13.190810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190788    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" containerName="payload-processing"
Nov 02 08:13:13.190924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190895    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" containerName="payload-processing"
Nov 02 08:13:13.190924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.190910    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d15ebc-b368-458e-a41a-f5b354fd98ef" containerName="payload-processing"
Nov 02 08:13:13.196029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.196010    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.219154 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.219116    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.219697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.219276    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.219697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.219426    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.219697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.219537    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtx2\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.222198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.222170    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:13.229207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229187    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"agc-cred-ea41b045e996\""
Nov 02 08:13:13.229357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229243    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c712b1\""
Nov 02 08:13:13.229436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229385    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"kube-root-ca.crt\""
Nov 02 08:13:13.229436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229429    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"routing-overlay\""
Nov 02 08:13:13.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229500    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-e2e--50b787fd-dockercfg-bpttw\""
Nov 02 08:13:13.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.229564    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"openshift-service-ca.crt\""
Nov 02 08:13:13.320870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.320832    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.321071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.320934    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.321071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.321012    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtx2\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.321071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.321069    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.321629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.321601    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.321807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.321787    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.323557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.323533    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.345036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.345006    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtx2\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.375463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.375432    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d15ebc-b368-458e-a41a-f5b354fd98ef" path="/var/lib/kubelet/pods/21d15ebc-b368-458e-a41a-f5b354fd98ef/volumes"
Nov 02 08:13:13.375994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.375977    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3726e13e-8216-4a83-a4bf-4e7ad4ca954f" path="/var/lib/kubelet/pods/3726e13e-8216-4a83-a4bf-4e7ad4ca954f/volumes"
Nov 02 08:13:13.510993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.510960    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:13.775476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.775403    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:13.829454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:13.829418    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:13.832633 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:13:13.832604    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2928fdd2_d09f_4fd2_a0ea_b3f4016041e9.slice/crio-15fbc715c637c916295eb758fb3aa13bc612499e9481df0757c7619657a86429 WatchSource:0}: Error finding container 15fbc715c637c916295eb758fb3aa13bc612499e9481df0757c7619657a86429: Status 404 returned error can't find the container with id 15fbc715c637c916295eb758fb3aa13bc612499e9481df0757c7619657a86429
Nov 02 08:13:14.297530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:14.292042    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" event={"ID":"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9","Type":"ContainerStarted","Data":"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"}
Nov 02 08:13:14.297530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:14.292117    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" event={"ID":"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9","Type":"ContainerStarted","Data":"15fbc715c637c916295eb758fb3aa13bc612499e9481df0757c7619657a86429"}
Nov 02 08:13:14.297530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:14.292301    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:14.357335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:14.348459    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" podStartSLOduration=1.3484381939999999 podStartE2EDuration="1.348438194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:14.347696454 +0000 UTC m=+2939.110133384" watchObservedRunningTime="2026-11-02 08:13:14.348438194 +0000 UTC m=+2939.110875101"
Nov 02 08:13:14.545883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:14.545847    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:13:17.725956 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:17.725915    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:13:19.293000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.292943    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" podUID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" containerName="payload-processing" containerID="cri-o://0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66" gracePeriod=25
Nov 02 08:13:19.587594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.587567    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:19.679139 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679106    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\") pod \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") "
Nov 02 08:13:19.679366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679176    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\") pod \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") "
Nov 02 08:13:19.679366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679272    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\") pod \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") "
Nov 02 08:13:19.679517 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679446    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\") pod \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\" (UID: \"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9\") "
Nov 02 08:13:19.679693 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679660    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume" pod "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" (UID: "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:19.679885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.679718    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume" pod "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" (UID: "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:19.680188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.680155    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.680286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.680190    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.681734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.681711    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2" pod "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" (UID: "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9"). InnerVolumeSpecName "kube-api-access-zqtx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:19.682007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.681992    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials" pod "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" (UID: "2928fdd2-d09f-4fd2-a0ea-b3f4016041e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:19.781709 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.781674    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtx2\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-kube-api-access-zqtx2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.781709 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:19.781705    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:20.367261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.367227    4515 generic.go:356] "Generic (PLEG): container finished" podID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" containerID="0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66" exitCode=0
Nov 02 08:13:20.367745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.367277    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"
Nov 02 08:13:20.367745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.367325    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" event={"ID":"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9","Type":"ContainerDied","Data":"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"}
Nov 02 08:13:20.367745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.367378    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls" event={"ID":"2928fdd2-d09f-4fd2-a0ea-b3f4016041e9","Type":"ContainerDied","Data":"15fbc715c637c916295eb758fb3aa13bc612499e9481df0757c7619657a86429"}
Nov 02 08:13:20.367745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.367401    4515 scope.go:122] "RemoveContainer" containerID="0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"
Nov 02 08:13:20.378758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.378735    4515 scope.go:122] "RemoveContainer" containerID="0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"
Nov 02 08:13:20.379050 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:20.379031    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66\": container with ID starting with 0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66 not found: ID does not exist" containerID="0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"
Nov 02 08:13:20.379109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.379061    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66"} err="failed to get container status \"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66\": rpc error: code = NotFound desc = could not find container \"0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66\": container with ID starting with 0daa763d52306055f44a6853bb0b2ce0537723efbb3c65dbd4bafbf644579c66 not found: ID does not exist"
Nov 02 08:13:20.433044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.432689    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:20.440803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:20.440764    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8759ls"]
Nov 02 08:13:21.375489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:21.375455    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" path="/var/lib/kubelet/pods/2928fdd2-d09f-4fd2-a0ea-b3f4016041e9/volumes"
Nov 02 08:13:28.458395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.458356    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:13:28.459147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.459128    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" containerName="payload-processing"
Nov 02 08:13:28.459189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.459148    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" containerName="payload-processing"
Nov 02 08:13:28.459266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.459257    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2928fdd2-d09f-4fd2-a0ea-b3f4016041e9" containerName="payload-processing"
Nov 02 08:13:28.465157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.465109    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.491581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.491446    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:13:28.497092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.497063    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e271b1-openshift-default-dockercfg-9t645\""
Nov 02 08:13:28.519557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519515    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519586    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74jz\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519660    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519723    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519808    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519851    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519889    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.519941 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.519922    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.520016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.520000    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621060    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621131    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621202    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621260    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621300    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621348    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621425    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621566    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621644    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74jz\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621677    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.621733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.621697    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.622424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.622153    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.622608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.622589    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.624775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.624754    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.625342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.625290    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.625760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.625451    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.676330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.672302    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74jz\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.694331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.688132    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\") pod \"e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") " pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:28.795334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:28.794598    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:29.138743 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:13:29.138709    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5be2cd_ab9c_4d7b_8c1c_ceec9caf782b.slice/crio-1de861bfdca6d0640c73c38e4d8e39f72993815b42d93c8464401edd6f7e8adf WatchSource:0}: Error finding container 1de861bfdca6d0640c73c38e4d8e39f72993815b42d93c8464401edd6f7e8adf: Status 404 returned error can't find the container with id 1de861bfdca6d0640c73c38e4d8e39f72993815b42d93c8464401edd6f7e8adf
Nov 02 08:13:29.140674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.140287    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:13:29.142969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.142922    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:13:29.143072 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.143022    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:13:29.143072 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.143062    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:13:29.433962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.433871    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerStarted","Data":"bf423c24f53a8991894b5e7ff29f570f40893a8369729f326890a9e4680d28da"}
Nov 02 08:13:29.433962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.433938    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerStarted","Data":"1de861bfdca6d0640c73c38e4d8e39f72993815b42d93c8464401edd6f7e8adf"}
Nov 02 08:13:29.489372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.489284    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podStartSLOduration=1.48927048 podStartE2EDuration="1.48927048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:29.48455695 +0000 UTC m=+2954.246993857" watchObservedRunningTime="2026-11-02 08:13:29.48927048 +0000 UTC m=+2954.251707436"
Nov 02 08:13:29.795268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:29.795230    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:30.803570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:30.803538    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:30.814676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:30.814644    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:13:31.447644 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:31.447615    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:31.448676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:31.448655    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:13:35.815702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:35.815656    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" podUID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" containerName="payload-processing" containerID="cri-o://b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780" gracePeriod=25
Nov 02 08:13:36.098706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.098681    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:13:36.183373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.183324    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\") pod \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") "
Nov 02 08:13:36.183588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.183403    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\") pod \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") "
Nov 02 08:13:36.183663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.183601    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\") pod \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") "
Nov 02 08:13:36.183729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.183669    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\") pod \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\" (UID: \"d2e65a5e-b150-431c-93af-7cf25d19cd4e\") "
Nov 02 08:13:36.183970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.183941    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume" pod "d2e65a5e-b150-431c-93af-7cf25d19cd4e" (UID: "d2e65a5e-b150-431c-93af-7cf25d19cd4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:36.184527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.184503    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:36.184972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.184949    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume" pod "d2e65a5e-b150-431c-93af-7cf25d19cd4e" (UID: "d2e65a5e-b150-431c-93af-7cf25d19cd4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:36.186461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.186423    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials" pod "d2e65a5e-b150-431c-93af-7cf25d19cd4e" (UID: "d2e65a5e-b150-431c-93af-7cf25d19cd4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:36.187430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.187405    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj" pod "d2e65a5e-b150-431c-93af-7cf25d19cd4e" (UID: "d2e65a5e-b150-431c-93af-7cf25d19cd4e"). InnerVolumeSpecName "kube-api-access-6hxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:36.285830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.285799    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:36.285830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.285828    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2e65a5e-b150-431c-93af-7cf25d19cd4e-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:36.285830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.285838    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxjj\" (UniqueName: \"kubernetes.io/projected/d2e65a5e-b150-431c-93af-7cf25d19cd4e-kube-api-access-6hxjj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:13:36.483807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.483723    4515 generic.go:356] "Generic (PLEG): container finished" podID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" containerID="b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780" exitCode=0
Nov 02 08:13:36.483807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.483774    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"
Nov 02 08:13:36.483807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.483793    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" event={"ID":"d2e65a5e-b150-431c-93af-7cf25d19cd4e","Type":"ContainerDied","Data":"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"}
Nov 02 08:13:36.484071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.483833    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt" event={"ID":"d2e65a5e-b150-431c-93af-7cf25d19cd4e","Type":"ContainerDied","Data":"b07ea0422e161effaeb45d5598baa239748d2960e66e3b8807d21c005427bf1a"}
Nov 02 08:13:36.484071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.483854    4515 scope.go:122] "RemoveContainer" containerID="b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"
Nov 02 08:13:36.495946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.495925    4515 scope.go:122] "RemoveContainer" containerID="b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"
Nov 02 08:13:36.496238 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:36.496215    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780\": container with ID starting with b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780 not found: ID does not exist" containerID="b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"
Nov 02 08:13:36.496283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.496247    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780"} err="failed to get container status \"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780\": rpc error: code = NotFound desc = could not find container \"b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780\": container with ID starting with b01bb465fe4cdb352b1eb0fdee9e52278155951717670d3e74bad2eac8971780 not found: ID does not exist"
Nov 02 08:13:36.538119 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.538083    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:13:36.541777 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:36.541751    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-56654vgwt"]
Nov 02 08:13:37.374699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:37.374664    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" path="/var/lib/kubelet/pods/d2e65a5e-b150-431c-93af-7cf25d19cd4e/volumes"
Nov 02 08:13:47.724473 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:13:47.724431    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:13:53.268507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.268359    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:13:53.269620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.269593    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" containerName="payload-processing"
Nov 02 08:13:53.269808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.269793    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" containerName="payload-processing"
Nov 02 08:13:53.274290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.270161    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e65a5e-b150-431c-93af-7cf25d19cd4e" containerName="payload-processing"
Nov 02 08:13:53.279352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.279278    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.288224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.288184    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.288443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.288303    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbvk\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.288536 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.288519    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.288594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.288575    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.289281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.289256    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:13:53.295473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.295443    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-d399c44b7e2e\""
Nov 02 08:13:53.296188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.296150    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:13:53.296334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.296236    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:13:53.296396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.296372    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:13:53.296450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.296435    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:13:53.296507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.296495    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-a89b1ea64886\""
Nov 02 08:13:53.322597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.307975    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-52g6v\""
Nov 02 08:13:53.408343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.404935    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.408343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.406011    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.408343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.406054    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbvk\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.408343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.406248    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.425336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.411346    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.425336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.405804    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.505901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.505830    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbvk\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.511080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.511042    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.605687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.605611    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:53.916499 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:13:53.916465    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8788fb76_0547_49a9_90bc_c945cef42e9a.slice/crio-be3417b7876b4fd28bc108b956a858f2199ee89f990172bf4bc70feb9716bb1e WatchSource:0}: Error finding container be3417b7876b4fd28bc108b956a858f2199ee89f990172bf4bc70feb9716bb1e: Status 404 returned error can't find the container with id be3417b7876b4fd28bc108b956a858f2199ee89f990172bf4bc70feb9716bb1e
Nov 02 08:13:53.917049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:53.917019    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:13:54.756855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:54.756810    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" event={"ID":"8788fb76-0547-49a9-90bc-c945cef42e9a","Type":"ContainerStarted","Data":"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"}
Nov 02 08:13:54.757503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:54.757474    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" event={"ID":"8788fb76-0547-49a9-90bc-c945cef42e9a","Type":"ContainerStarted","Data":"be3417b7876b4fd28bc108b956a858f2199ee89f990172bf4bc70feb9716bb1e"}
Nov 02 08:13:54.758753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:54.758730    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:13:54.828335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:54.822354    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" podStartSLOduration=1.822329285 podStartE2EDuration="1.822329285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:54.80927324 +0000 UTC m=+2979.571710147" watchObservedRunningTime="2026-11-02 08:13:54.822329285 +0000 UTC m=+2979.584766263"
Nov 02 08:13:56.338954 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.338919    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:13:56.343653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.343629    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.372420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372387    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:13:56.372653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372418    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-f45c1914cd99\""
Nov 02 08:13:56.372653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372437    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:13:56.372653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372460    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-b8dc688374ca\""
Nov 02 08:13:56.372927 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372495    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:13:56.372927 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.372540    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:13:56.373051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.373006    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-hftbt\""
Nov 02 08:13:56.377269 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.377244    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:13:56.385043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.385014    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.385225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.385084    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcqv\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.385326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.385261    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.385413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.385369    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.487133 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.487093    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.487386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.487184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.487386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.487287    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.487531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.487414    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcqv\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.488053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.488026    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.488174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.488133    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.490021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.489998    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.511462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.511432    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcqv\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.659522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.659434    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:56.967047 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:13:56.967017    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4839b5_0ab8_4fb4_b923_c81860b061d7.slice/crio-ef8894dcc5aee0539f8296e990b01c678a896c47cf0ef15e29415cafd2ccd0b6 WatchSource:0}: Error finding container ef8894dcc5aee0539f8296e990b01c678a896c47cf0ef15e29415cafd2ccd0b6: Status 404 returned error can't find the container with id ef8894dcc5aee0539f8296e990b01c678a896c47cf0ef15e29415cafd2ccd0b6
Nov 02 08:13:56.969754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:56.969729    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:13:57.780417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:57.780379    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" event={"ID":"7e4839b5-0ab8-4fb4-b923-c81860b061d7","Type":"ContainerStarted","Data":"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"}
Nov 02 08:13:57.780935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:57.780444    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" event={"ID":"7e4839b5-0ab8-4fb4-b923-c81860b061d7","Type":"ContainerStarted","Data":"ef8894dcc5aee0539f8296e990b01c678a896c47cf0ef15e29415cafd2ccd0b6"}
Nov 02 08:13:57.780935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:57.780476    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:13:57.824833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:13:57.824775    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" podStartSLOduration=1.824761331 podStartE2EDuration="1.824761331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:57.821901198 +0000 UTC m=+2982.584338104" watchObservedRunningTime="2026-11-02 08:13:57.824761331 +0000 UTC m=+2982.587198237"
Nov 02 08:14:06.775705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:06.775672    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:14:08.825541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:08.825511    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:14:11.446112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:11.446077    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:14:12.134288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.134256    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:14:12.140146 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.140122    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.156283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.156254    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:14:12.167366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.164201    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c4489d\""
Nov 02 08:14:12.167366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.164238    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c4489d-dockercfg-h9zxg\""
Nov 02 08:14:12.178321 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.178273    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.178538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.178517    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxh5\" (UniqueName: \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.316468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.315163    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxh5\" (UniqueName: \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.316468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.315444    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.316735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.316507    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.398985 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.398868    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxh5\" (UniqueName: \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\") pod \"payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.461594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.461560    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:12.841736 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:14:12.841696    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb163df41_7bca_4a0f_8e3f_5153e0aaa86e.slice/crio-ad589099eb41125cf3213f2975b0f518b307da41ebeaa68a4a2c45db9b7ca265 WatchSource:0}: Error finding container ad589099eb41125cf3213f2975b0f518b307da41ebeaa68a4a2c45db9b7ca265: Status 404 returned error can't find the container with id ad589099eb41125cf3213f2975b0f518b307da41ebeaa68a4a2c45db9b7ca265
Nov 02 08:14:12.842858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.842833    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:14:12.890003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:12.889974    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" event={"ID":"b163df41-7bca-4a0f-8e3f-5153e0aaa86e","Type":"ContainerStarted","Data":"ad589099eb41125cf3213f2975b0f518b307da41ebeaa68a4a2c45db9b7ca265"}
Nov 02 08:14:13.899264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:13.899224    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" event={"ID":"b163df41-7bca-4a0f-8e3f-5153e0aaa86e","Type":"ContainerStarted","Data":"38afe07170b254ff0d2b3fc3db8a4aa74ed4e03b02fcbef61c3bf2ebe46d9b56"}
Nov 02 08:14:13.899739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:13.899344    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:13.944413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:13.944359    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" podStartSLOduration=1.944342883 podStartE2EDuration="1.944342883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:13.941289361 +0000 UTC m=+2998.703726266" watchObservedRunningTime="2026-11-02 08:14:13.944342883 +0000 UTC m=+2998.706779789"
Nov 02 08:14:17.725184 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:14:17.725145    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:14:24.910152 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:24.910117    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:14:47.747991 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:14:47.747947    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:14:54.852003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:54.851966    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:14:54.857198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:54.857173    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:54.880471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:54.880406    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:14:54.886413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:54.886387    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c4489d-serving-cert\""
Nov 02 08:14:55.013898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.013857    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwg87\" (UniqueName: \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.014112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.014027    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.014112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.014064    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.014232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.014214    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.116299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.116205    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwg87\" (UniqueName: \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.116498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.116387    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.116498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.116424    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.116606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.116522    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.117398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.117372    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.119184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.119157    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.119401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.119378    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.140301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.140270    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwg87\" (UniqueName: \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\") pod \"maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.186080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.185803    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:55.533733 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:14:55.533701    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772f51c2_91cb_4407_8cd9_ff54af72c83f.slice/crio-b6e3b8dab92b03dd5e26690af1466fb23d53325e9be675d1cc49ac28ba330ae6 WatchSource:0}: Error finding container b6e3b8dab92b03dd5e26690af1466fb23d53325e9be675d1cc49ac28ba330ae6: Status 404 returned error can't find the container with id b6e3b8dab92b03dd5e26690af1466fb23d53325e9be675d1cc49ac28ba330ae6
Nov 02 08:14:55.534556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.534531    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:14:55.535270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:55.535250    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 08:14:56.329839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.329800    4515 generic.go:356] "Generic (PLEG): container finished" podID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerID="bf423c24f53a8991894b5e7ff29f570f40893a8369729f326890a9e4680d28da" exitCode=0
Nov 02 08:14:56.330360 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.329870    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerDied","Data":"bf423c24f53a8991894b5e7ff29f570f40893a8369729f326890a9e4680d28da"}
Nov 02 08:14:56.330447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.330413    4515 scope.go:122] "RemoveContainer" containerID="bf423c24f53a8991894b5e7ff29f570f40893a8369729f326890a9e4680d28da"
Nov 02 08:14:56.331844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.331818    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" event={"ID":"772f51c2-91cb-4407-8cd9-ff54af72c83f","Type":"ContainerStarted","Data":"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"}
Nov 02 08:14:56.331965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.331878    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" event={"ID":"772f51c2-91cb-4407-8cd9-ff54af72c83f","Type":"ContainerStarted","Data":"b6e3b8dab92b03dd5e26690af1466fb23d53325e9be675d1cc49ac28ba330ae6"}
Nov 02 08:14:56.332050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.331986    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:14:56.332564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.332524    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:56.332657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.332605    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:56.332657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.332639    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:56.432530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:56.432295    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" podStartSLOduration=2.05855311 podStartE2EDuration="2.432276346s" totalImagesPullingTime="373.723236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:56.431767529 +0000 UTC m=+3041.194204441" watchObservedRunningTime="2026-11-02 08:14:56.432276346 +0000 UTC m=+3041.194713259"
Nov 02 08:14:57.347793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:57.347751    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerStarted","Data":"c37389f33e77f55bffbfff0dc1c8d25092d2d53eca4612832dd3ae671280013b"}
Nov 02 08:14:57.795322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:57.795272    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:14:57.796450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:57.796413    4515 patch_prober.go:29] interesting pod/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 02 08:14:57.796589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:57.796496    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 02 08:14:58.796178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:58.796139    4515 patch_prober.go:29] interesting pod/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 02 08:14:58.796702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:58.796205    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 02 08:14:59.796318 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:59.796284    4515 patch_prober.go:29] interesting pod/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 02 08:14:59.796747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:14:59.796375    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 02 08:15:00.182422 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.182344    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:00.186916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.186890    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.213077 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.213014    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:00.260275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.260243    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.283410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.283373    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:00.285712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.285681    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:15:00.292394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.292367    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:00.299171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.299145    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.305437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.303618    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:00.308760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.308731    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.308986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.308892    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcj4\" (UniqueName: \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.309977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.309937    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.310527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.310502    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:00.358105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.358072    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:00.382685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.359629    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.396812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.396433    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:00.405021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.404991    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:00.407712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.407688    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:00.410370 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410347    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.410518 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410407    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcj4\" (UniqueName: \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.410518 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410450    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6ph\" (UniqueName: \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.410518 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410483    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410564    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5dc\" (UniqueName: \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410654    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj5m\" (UniqueName: \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410790    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410821    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410857    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzgg\" (UniqueName: \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.411491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.410994    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.412220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.412193    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.412524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.412503    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.413546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.413527    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:00.423457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.423424    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:00.460884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.460598    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcj4\" (UniqueName: \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.461346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.461297    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.478722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.478691    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:00.488753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.488728    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:00.488950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.488848    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.489266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.489244    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:00.495260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.495238    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.496561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.496537    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:00.500188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.500169    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:00.509924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.509881    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:00.510057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.509955    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:00.512596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512573    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.512683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512643    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.512729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512706    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbtw\" (UniqueName: \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.513203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512738    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6ph\" (UniqueName: \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.513203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512767    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.513203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.512833    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5dc\" (UniqueName: \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.513455 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513422    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj5m\" (UniqueName: \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.513523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513486    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.513523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513494    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.513640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513529    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.513704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513633    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.513704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513671    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzgg\" (UniqueName: \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.513704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513675    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.514037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.513698    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.514111 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.514064    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qkr\" (UniqueName: \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.514570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.514551    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.547610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.546842    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6ph\" (UniqueName: \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.547610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.547110    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzgg\" (UniqueName: \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.547610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.547445    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj5m\" (UniqueName: \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.548208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.548185    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5dc\" (UniqueName: \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.601203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.601180    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615585    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615660    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615769    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615834    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qkr\" (UniqueName: \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615909    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615958    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjff\" (UniqueName: \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.615992    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbtw\" (UniqueName: \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.616176    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.616992    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.642347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.617688    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.655462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.655212    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qkr\" (UniqueName: \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.655611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.655581    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbtw\" (UniqueName: \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\") pod \"maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.661627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.661599    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:00.697721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.697450    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:00.713512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.713432    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:00.751513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.750426    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.751513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.750565    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.751513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.750638    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjff\" (UniqueName: \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.751513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.750848    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.751513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.751246    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.752844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.752817    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.824358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.815531    4515 patch_prober.go:29] interesting pod/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 02 08:15:00.824358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.815589    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 02 08:15:00.856362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.835268    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:00.856362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.846885    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:00.856362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.854141    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjff\" (UniqueName: \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\") pod \"maas-api-key-cleanup-29893455-6dp2g\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:00.869543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.869380    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:00.873415 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:00.873381    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03855a29_10d7_4364_a77c_7c15dbea89ed.slice/crio-b3e527255023f0ef99d4b3e316983f8ec940cfe5f8fd6ecacc6c1a07b03e0cbd WatchSource:0}: Error finding container b3e527255023f0ef99d4b3e316983f8ec940cfe5f8fd6ecacc6c1a07b03e0cbd: Status 404 returned error can't find the container with id b3e527255023f0ef99d4b3e316983f8ec940cfe5f8fd6ecacc6c1a07b03e0cbd
Nov 02 08:15:00.873959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:00.873918    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:01.105409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.105284    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:01.116076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.116043    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:01.137676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.137192    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:01.151889 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.151839    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fd1c09_42ca_4dc9_ac24_c3d8fb023bcc.slice/crio-ec4df5110eea4cab10abb659a238c5f6a8038d97e262733c16ce4e404895b8f4 WatchSource:0}: Error finding container ec4df5110eea4cab10abb659a238c5f6a8038d97e262733c16ce4e404895b8f4: Status 404 returned error can't find the container with id ec4df5110eea4cab10abb659a238c5f6a8038d97e262733c16ce4e404895b8f4
Nov 02 08:15:01.200879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.200843    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:01.203508 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.202317    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3036233b_9580_4919_a9b7_151df8375612.slice/crio-2688f22e24cae6a486f47761bdea409ae4f22dcdd3e66df459861da625ec07e6 WatchSource:0}: Error finding container 2688f22e24cae6a486f47761bdea409ae4f22dcdd3e66df459861da625ec07e6: Status 404 returned error can't find the container with id 2688f22e24cae6a486f47761bdea409ae4f22dcdd3e66df459861da625ec07e6
Nov 02 08:15:01.248810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.223759    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:01.271855 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.271808    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4857be_3490_4bdc_8dee_7706fee266cb.slice/crio-fc07b1a93a8e2a8d6eaa35adabef5e9f46034b6867882b46829cc20382fc9eec WatchSource:0}: Error finding container fc07b1a93a8e2a8d6eaa35adabef5e9f46034b6867882b46829cc20382fc9eec: Status 404 returned error can't find the container with id fc07b1a93a8e2a8d6eaa35adabef5e9f46034b6867882b46829cc20382fc9eec
Nov 02 08:15:01.386071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.385997    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerStarted","Data":"ec4df5110eea4cab10abb659a238c5f6a8038d97e262733c16ce4e404895b8f4"}
Nov 02 08:15:01.387642 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.387607    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerStarted","Data":"2688f22e24cae6a486f47761bdea409ae4f22dcdd3e66df459861da625ec07e6"}
Nov 02 08:15:01.389164 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.389135    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" event={"ID":"03855a29-10d7-4364-a77c-7c15dbea89ed","Type":"ContainerStarted","Data":"b3e527255023f0ef99d4b3e316983f8ec940cfe5f8fd6ecacc6c1a07b03e0cbd"}
Nov 02 08:15:01.390718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.390685    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerStarted","Data":"fc07b1a93a8e2a8d6eaa35adabef5e9f46034b6867882b46829cc20382fc9eec"}
Nov 02 08:15:01.449070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.447610    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:15:01.452744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.451517    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:01.458562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.458525    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:01.502753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.502723    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:01.552234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.552197    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:01.555679 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.555331    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a653e6_e7e0_4fd1_8b0e_7d46d09ef88d.slice/crio-8a1bb620f06b5c370e6764ed2a0d1d762deb7e733f7d3f68e22b2b332f0ce1d6 WatchSource:0}: Error finding container 8a1bb620f06b5c370e6764ed2a0d1d762deb7e733f7d3f68e22b2b332f0ce1d6: Status 404 returned error can't find the container with id 8a1bb620f06b5c370e6764ed2a0d1d762deb7e733f7d3f68e22b2b332f0ce1d6
Nov 02 08:15:01.556437 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.556387    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b51c3bf_8bbe_4380_b2e3_c63e6555aae2.slice/crio-c9a1610906454b6907491d5010878230d695d6167dcac1709185c4f1f462cbf9 WatchSource:0}: Error finding container c9a1610906454b6907491d5010878230d695d6167dcac1709185c4f1f462cbf9: Status 404 returned error can't find the container with id c9a1610906454b6907491d5010878230d695d6167dcac1709185c4f1f462cbf9
Nov 02 08:15:01.557938 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:01.557916    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828dc10b_98ae_4ea9_95fc_b34a6fd6447c.slice/crio-a11d7cdbe0b2025d0b553241bc11e00f9970d20b126e2a74b67b2ff720994b34 WatchSource:0}: Error finding container a11d7cdbe0b2025d0b553241bc11e00f9970d20b126e2a74b67b2ff720994b34: Status 404 returned error can't find the container with id a11d7cdbe0b2025d0b553241bc11e00f9970d20b126e2a74b67b2ff720994b34
Nov 02 08:15:01.558407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.558386    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:01.795630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.795544    4515 patch_prober.go:29] interesting pod/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 02 08:15:01.795630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:01.795601    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 02 08:15:02.354253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.354223    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:15:02.400135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.400034    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerStarted","Data":"ade4ab0086b5274133ab8148a8564446f1a4760e8d5496960b4c409dbafe5470"}
Nov 02 08:15:02.400135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.400093    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerStarted","Data":"80c227a307ddc176805d8be1ddc986e2189b42a072fcdc5e82b3782d0705ee22"}
Nov 02 08:15:02.402547 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.402516    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerStarted","Data":"575e721568ea5d8f7f9f8e08b41e65af8f2718e8c2dc4527b1c85952cf345a0e"}
Nov 02 08:15:02.402676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.402581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerStarted","Data":"234f75c80a45936d6089bc8668b712cd94a1b53edd92524bc235f54c03710300"}
Nov 02 08:15:02.404300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.404274    4515 generic.go:356] "Generic (PLEG): container finished" podID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerID="431eeb9162942cc547c67598b8f2d82b0cf2b0238fb08800d51c4ba9cdf04493" exitCode=60
Nov 02 08:15:02.404427 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.404383    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" event={"ID":"03855a29-10d7-4364-a77c-7c15dbea89ed","Type":"ContainerDied","Data":"431eeb9162942cc547c67598b8f2d82b0cf2b0238fb08800d51c4ba9cdf04493"}
Nov 02 08:15:02.404652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.404629    4515 scope.go:122] "RemoveContainer" containerID="431eeb9162942cc547c67598b8f2d82b0cf2b0238fb08800d51c4ba9cdf04493"
Nov 02 08:15:02.408856 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.408828    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerStarted","Data":"9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c"}
Nov 02 08:15:02.408920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.408870    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerStarted","Data":"c9a1610906454b6907491d5010878230d695d6167dcac1709185c4f1f462cbf9"}
Nov 02 08:15:02.410772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.410724    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerStarted","Data":"43aa0a36fab9821a855188e54e7ca5cbe5b14f4dc7aaf8b581bc5f16591ea8e7"}
Nov 02 08:15:02.412379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.412358    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerStarted","Data":"1cf9aa8bb301f632b90062b46bbe52389fb7a4340c4c8f0ab5794d35c001e479"}
Nov 02 08:15:02.414432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.414409    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerStarted","Data":"286604fae6d8d50b364c22d715492ec1501d51961b019870c666ce6cc6a80624"}
Nov 02 08:15:02.414599 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.414581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerStarted","Data":"a11d7cdbe0b2025d0b553241bc11e00f9970d20b126e2a74b67b2ff720994b34"}
Nov 02 08:15:02.420403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.420371    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerStarted","Data":"8bf9c9949379fa338338281cd542db3c094c9688e179b8a4fa280ce9b6cd655e"}
Nov 02 08:15:02.420568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.420552    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerStarted","Data":"8a1bb620f06b5c370e6764ed2a0d1d762deb7e733f7d3f68e22b2b332f0ce1d6"}
Nov 02 08:15:02.452181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.452140    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerStarted","Data":"14df20698ba5bb4cfe5f709e9c5d580b285c88f2859d46ce721e44dde79ad456"}
Nov 02 08:15:02.454786 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.454630    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" podStartSLOduration=2.454612329 podStartE2EDuration="2.454612329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.450959221 +0000 UTC m=+3047.213396129" watchObservedRunningTime="2026-11-02 08:15:02.454612329 +0000 UTC m=+3047.217049236"
Nov 02 08:15:02.495225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.495156    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" podStartSLOduration=2.4951348749999998 podStartE2EDuration="2.495134875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.492800625 +0000 UTC m=+3047.255237532" watchObservedRunningTime="2026-11-02 08:15:02.495134875 +0000 UTC m=+3047.257571826"
Nov 02 08:15:02.544507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.543458    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" podStartSLOduration=2.543437391 podStartE2EDuration="2.543437391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.537645513 +0000 UTC m=+3047.300082420" watchObservedRunningTime="2026-11-02 08:15:02.543437391 +0000 UTC m=+3047.305874297"
Nov 02 08:15:02.590494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.590440    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" podStartSLOduration=2.5904266959999998 podStartE2EDuration="2.590426696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.587727662 +0000 UTC m=+3047.350164571" watchObservedRunningTime="2026-11-02 08:15:02.590426696 +0000 UTC m=+3047.352863603"
Nov 02 08:15:02.630497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.630433    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" podStartSLOduration=2.630413637 podStartE2EDuration="2.630413637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.630135493 +0000 UTC m=+3047.392572400" watchObservedRunningTime="2026-11-02 08:15:02.630413637 +0000 UTC m=+3047.392850545"
Nov 02 08:15:02.676010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.675666    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" podStartSLOduration=2.675643262 podStartE2EDuration="2.675643262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.669949694 +0000 UTC m=+3047.432386601" watchObservedRunningTime="2026-11-02 08:15:02.675643262 +0000 UTC m=+3047.438080165"
Nov 02 08:15:02.779369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.777781    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" podStartSLOduration=2.777759301 podStartE2EDuration="2.777759301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.771872057 +0000 UTC m=+3047.534308992" watchObservedRunningTime="2026-11-02 08:15:02.777759301 +0000 UTC m=+3047.540196208"
Nov 02 08:15:02.806534 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.806500    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:15:02.807790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.807767    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:15:02.815268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:02.815197    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" podStartSLOduration=2.815180394 podStartE2EDuration="2.815180394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.813597899 +0000 UTC m=+3047.576034806" watchObservedRunningTime="2026-11-02 08:15:02.815180394 +0000 UTC m=+3047.577617304"
Nov 02 08:15:02.942549 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:02.942472    4515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b51c3bf_8bbe_4380_b2e3_c63e6555aae2.slice/crio-conmon-9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 08:15:03.461693 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.461658    4515 generic.go:356] "Generic (PLEG): container finished" podID="3036233b-9580-4919-a9b7-151df8375612" containerID="1cf9aa8bb301f632b90062b46bbe52389fb7a4340c4c8f0ab5794d35c001e479" exitCode=60
Nov 02 08:15:03.462144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.461742    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerDied","Data":"1cf9aa8bb301f632b90062b46bbe52389fb7a4340c4c8f0ab5794d35c001e479"}
Nov 02 08:15:03.462144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.462072    4515 scope.go:122] "RemoveContainer" containerID="1cf9aa8bb301f632b90062b46bbe52389fb7a4340c4c8f0ab5794d35c001e479"
Nov 02 08:15:03.463680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.463659    4515 generic.go:356] "Generic (PLEG): container finished" podID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerID="286604fae6d8d50b364c22d715492ec1501d51961b019870c666ce6cc6a80624" exitCode=60
Nov 02 08:15:03.463785 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.463746    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerDied","Data":"286604fae6d8d50b364c22d715492ec1501d51961b019870c666ce6cc6a80624"}
Nov 02 08:15:03.464002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.463984    4515 scope.go:122] "RemoveContainer" containerID="286604fae6d8d50b364c22d715492ec1501d51961b019870c666ce6cc6a80624"
Nov 02 08:15:03.465867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.465753    4515 generic.go:356] "Generic (PLEG): container finished" podID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerID="8bf9c9949379fa338338281cd542db3c094c9688e179b8a4fa280ce9b6cd655e" exitCode=60
Nov 02 08:15:03.465867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.465835    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerDied","Data":"8bf9c9949379fa338338281cd542db3c094c9688e179b8a4fa280ce9b6cd655e"}
Nov 02 08:15:03.466143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.466125    4515 scope.go:122] "RemoveContainer" containerID="8bf9c9949379fa338338281cd542db3c094c9688e179b8a4fa280ce9b6cd655e"
Nov 02 08:15:03.467859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.467689    4515 generic.go:356] "Generic (PLEG): container finished" podID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerID="14df20698ba5bb4cfe5f709e9c5d580b285c88f2859d46ce721e44dde79ad456" exitCode=60
Nov 02 08:15:03.467859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.467746    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerDied","Data":"14df20698ba5bb4cfe5f709e9c5d580b285c88f2859d46ce721e44dde79ad456"}
Nov 02 08:15:03.468027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.467951    4515 scope.go:122] "RemoveContainer" containerID="14df20698ba5bb4cfe5f709e9c5d580b285c88f2859d46ce721e44dde79ad456"
Nov 02 08:15:03.469492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.469470    4515 generic.go:356] "Generic (PLEG): container finished" podID="f07106ac-f7a0-4029-b901-58387cb85744" containerID="ade4ab0086b5274133ab8148a8564446f1a4760e8d5496960b4c409dbafe5470" exitCode=60
Nov 02 08:15:03.469578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.469554    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerDied","Data":"ade4ab0086b5274133ab8148a8564446f1a4760e8d5496960b4c409dbafe5470"}
Nov 02 08:15:03.469757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.469743    4515 scope.go:122] "RemoveContainer" containerID="ade4ab0086b5274133ab8148a8564446f1a4760e8d5496960b4c409dbafe5470"
Nov 02 08:15:03.471098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.471080    4515 generic.go:356] "Generic (PLEG): container finished" podID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerID="575e721568ea5d8f7f9f8e08b41e65af8f2718e8c2dc4527b1c85952cf345a0e" exitCode=60
Nov 02 08:15:03.471218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.471139    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerDied","Data":"575e721568ea5d8f7f9f8e08b41e65af8f2718e8c2dc4527b1c85952cf345a0e"}
Nov 02 08:15:03.471418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.471400    4515 scope.go:122] "RemoveContainer" containerID="575e721568ea5d8f7f9f8e08b41e65af8f2718e8c2dc4527b1c85952cf345a0e"
Nov 02 08:15:03.473700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.473679    4515 generic.go:356] "Generic (PLEG): container finished" podID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerID="cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0" exitCode=60
Nov 02 08:15:03.473803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.473788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" event={"ID":"03855a29-10d7-4364-a77c-7c15dbea89ed","Type":"ContainerDied","Data":"cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0"}
Nov 02 08:15:03.473852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.473839    4515 scope.go:122] "RemoveContainer" containerID="431eeb9162942cc547c67598b8f2d82b0cf2b0238fb08800d51c4ba9cdf04493"
Nov 02 08:15:03.473958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.473944    4515 scope.go:122] "RemoveContainer" containerID="cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0"
Nov 02 08:15:03.474291 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:03.474232    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp_odh-ai-gateway-infra(03855a29-10d7-4364-a77c-7c15dbea89ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed"
Nov 02 08:15:03.476718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.476695    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerID="9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c" exitCode=60
Nov 02 08:15:03.476818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.476782    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerDied","Data":"9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c"}
Nov 02 08:15:03.477056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.477040    4515 scope.go:122] "RemoveContainer" containerID="9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c"
Nov 02 08:15:03.478763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.478733    4515 generic.go:356] "Generic (PLEG): container finished" podID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerID="43aa0a36fab9821a855188e54e7ca5cbe5b14f4dc7aaf8b581bc5f16591ea8e7" exitCode=60
Nov 02 08:15:03.478841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.478788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerDied","Data":"43aa0a36fab9821a855188e54e7ca5cbe5b14f4dc7aaf8b581bc5f16591ea8e7"}
Nov 02 08:15:03.479011 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:03.478995    4515 scope.go:122] "RemoveContainer" containerID="43aa0a36fab9821a855188e54e7ca5cbe5b14f4dc7aaf8b581bc5f16591ea8e7"
Nov 02 08:15:04.490025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.489989    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerStarted","Data":"6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"}
Nov 02 08:15:04.492165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.492138    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerStarted","Data":"83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"}
Nov 02 08:15:04.494259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.494219    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerStarted","Data":"7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"}
Nov 02 08:15:04.496791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.496762    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerStarted","Data":"b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"}
Nov 02 08:15:04.498975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.498938    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerStarted","Data":"6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"}
Nov 02 08:15:04.500958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.500928    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerStarted","Data":"99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"}
Nov 02 08:15:04.503064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.503043    4515 scope.go:122] "RemoveContainer" containerID="cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0"
Nov 02 08:15:04.503396 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:04.503359    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp_odh-ai-gateway-infra(03855a29-10d7-4364-a77c-7c15dbea89ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed"
Nov 02 08:15:04.505000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.504976    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerStarted","Data":"363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"}
Nov 02 08:15:04.506826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:04.506806    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerStarted","Data":"63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"}
Nov 02 08:15:05.517073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.517035    4515 generic.go:356] "Generic (PLEG): container finished" podID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerID="83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760" exitCode=60
Nov 02 08:15:05.517564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.517129    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerDied","Data":"83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"}
Nov 02 08:15:05.517564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.517181    4515 scope.go:122] "RemoveContainer" containerID="286604fae6d8d50b364c22d715492ec1501d51961b019870c666ce6cc6a80624"
Nov 02 08:15:05.517564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.517479    4515 scope.go:122] "RemoveContainer" containerID="83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"
Nov 02 08:15:05.517909 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.517873    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr_odh-ai-gateway-infra(828dc10b-98ae-4ea9-95fc-b34a6fd6447c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c"
Nov 02 08:15:05.519363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.519281    4515 generic.go:356] "Generic (PLEG): container finished" podID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerID="7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9" exitCode=60
Nov 02 08:15:05.519469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.519355    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerDied","Data":"7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"}
Nov 02 08:15:05.519699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.519682    4515 scope.go:122] "RemoveContainer" containerID="7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"
Nov 02 08:15:05.520044 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.520016    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz_odh-ai-gateway-infra(46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d"
Nov 02 08:15:05.521363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.521339    4515 generic.go:356] "Generic (PLEG): container finished" podID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerID="b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae" exitCode=60
Nov 02 08:15:05.521463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.521417    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerDied","Data":"b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"}
Nov 02 08:15:05.521701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.521683    4515 scope.go:122] "RemoveContainer" containerID="b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"
Nov 02 08:15:05.522028 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.522007    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq_odh-ai-gateway-infra(c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc"
Nov 02 08:15:05.523560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.523539    4515 generic.go:356] "Generic (PLEG): container finished" podID="f07106ac-f7a0-4029-b901-58387cb85744" containerID="6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8" exitCode=60
Nov 02 08:15:05.523667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.523615    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerDied","Data":"6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"}
Nov 02 08:15:05.523864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.523848    4515 scope.go:122] "RemoveContainer" containerID="6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"
Nov 02 08:15:05.524174 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.524155    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg_odh-ai-gateway-infra(f07106ac-f7a0-4029-b901-58387cb85744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" podUID="f07106ac-f7a0-4029-b901-58387cb85744"
Nov 02 08:15:05.525670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.525647    4515 generic.go:356] "Generic (PLEG): container finished" podID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerID="99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b" exitCode=60
Nov 02 08:15:05.525876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.525856    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerDied","Data":"99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"}
Nov 02 08:15:05.525974 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.525955    4515 scope.go:122] "RemoveContainer" containerID="99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"
Nov 02 08:15:05.526286 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.526258    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-6dp2g_odh-ai-gateway-infra(707314c1-4961-42b4-94c0-dad02c5ad2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe"
Nov 02 08:15:05.527649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.527629    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerID="363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89" exitCode=60
Nov 02 08:15:05.527738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.527706    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerDied","Data":"363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"}
Nov 02 08:15:05.527988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.527956    4515 scope.go:122] "RemoveContainer" containerID="363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"
Nov 02 08:15:05.528296 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.528272    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6_odh-ai-gateway-infra(1b51c3bf-8bbe-4380-b2e3-c63e6555aae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2"
Nov 02 08:15:05.529689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.529669    4515 generic.go:356] "Generic (PLEG): container finished" podID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerID="63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607" exitCode=60
Nov 02 08:15:05.529795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.529764    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerDied","Data":"63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"}
Nov 02 08:15:05.530058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.530040    4515 scope.go:122] "RemoveContainer" containerID="63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"
Nov 02 08:15:05.530388 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.530364    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw_odh-ai-gateway-infra(4d4857be-3490-4bdc-8dee-7706fee266cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb"
Nov 02 08:15:05.532931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.532907    4515 generic.go:356] "Generic (PLEG): container finished" podID="3036233b-9580-4919-a9b7-151df8375612" containerID="6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51" exitCode=60
Nov 02 08:15:05.533050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.532959    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerDied","Data":"6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"}
Nov 02 08:15:05.533184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.533078    4515 scope.go:122] "RemoveContainer" containerID="8bf9c9949379fa338338281cd542db3c094c9688e179b8a4fa280ce9b6cd655e"
Nov 02 08:15:05.533184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.533171    4515 scope.go:122] "RemoveContainer" containerID="6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"
Nov 02 08:15:05.533448 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:05.533430    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx_odh-ai-gateway-infra(3036233b-9580-4919-a9b7-151df8375612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" podUID="3036233b-9580-4919-a9b7-151df8375612"
Nov 02 08:15:05.547554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.547531    4515 scope.go:122] "RemoveContainer" containerID="14df20698ba5bb4cfe5f709e9c5d580b285c88f2859d46ce721e44dde79ad456"
Nov 02 08:15:05.562161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.562135    4515 scope.go:122] "RemoveContainer" containerID="ade4ab0086b5274133ab8148a8564446f1a4760e8d5496960b4c409dbafe5470"
Nov 02 08:15:05.576572 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.576553    4515 scope.go:122] "RemoveContainer" containerID="575e721568ea5d8f7f9f8e08b41e65af8f2718e8c2dc4527b1c85952cf345a0e"
Nov 02 08:15:05.590897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.590876    4515 scope.go:122] "RemoveContainer" containerID="9cd5112ca9d11f6e6f2f36ee928e6b2ebb88530d4229d84cae753af4a680d13c"
Nov 02 08:15:05.605212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.605189    4515 scope.go:122] "RemoveContainer" containerID="43aa0a36fab9821a855188e54e7ca5cbe5b14f4dc7aaf8b581bc5f16591ea8e7"
Nov 02 08:15:05.619890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:05.619862    4515 scope.go:122] "RemoveContainer" containerID="1cf9aa8bb301f632b90062b46bbe52389fb7a4340c4c8f0ab5794d35c001e479"
Nov 02 08:15:06.582110 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.582065    4515 scope.go:122] "RemoveContainer" containerID="363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"
Nov 02 08:15:06.582827 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.582678    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6_odh-ai-gateway-infra(1b51c3bf-8bbe-4380-b2e3-c63e6555aae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2"
Nov 02 08:15:06.584610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.584584    4515 scope.go:122] "RemoveContainer" containerID="63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"
Nov 02 08:15:06.585122 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.585090    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw_odh-ai-gateway-infra(4d4857be-3490-4bdc-8dee-7706fee266cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb"
Nov 02 08:15:06.586516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.586495    4515 scope.go:122] "RemoveContainer" containerID="6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"
Nov 02 08:15:06.586779 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.586756    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx_odh-ai-gateway-infra(3036233b-9580-4919-a9b7-151df8375612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" podUID="3036233b-9580-4919-a9b7-151df8375612"
Nov 02 08:15:06.588687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.588666    4515 scope.go:122] "RemoveContainer" containerID="83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"
Nov 02 08:15:06.588968 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.588938    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr_odh-ai-gateway-infra(828dc10b-98ae-4ea9-95fc-b34a6fd6447c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c"
Nov 02 08:15:06.590659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.590632    4515 scope.go:122] "RemoveContainer" containerID="7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"
Nov 02 08:15:06.590872 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.590843    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz_odh-ai-gateway-infra(46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d"
Nov 02 08:15:06.592442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.592424    4515 scope.go:122] "RemoveContainer" containerID="b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"
Nov 02 08:15:06.592666 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.592644    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq_odh-ai-gateway-infra(c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc"
Nov 02 08:15:06.594211 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.594191    4515 scope.go:122] "RemoveContainer" containerID="6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"
Nov 02 08:15:06.594475 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.594451    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg_odh-ai-gateway-infra(f07106ac-f7a0-4029-b901-58387cb85744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" podUID="f07106ac-f7a0-4029-b901-58387cb85744"
Nov 02 08:15:06.595958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:06.595934    4515 scope.go:122] "RemoveContainer" containerID="99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"
Nov 02 08:15:06.596213 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:06.596189    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-6dp2g_odh-ai-gateway-infra(707314c1-4961-42b4-94c0-dad02c5ad2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe"
Nov 02 08:15:09.909365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:09.909328    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:15:15.371730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.371700    4515 scope.go:122] "RemoveContainer" containerID="b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"
Nov 02 08:15:15.372260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.371958    4515 scope.go:122] "RemoveContainer" containerID="cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0"
Nov 02 08:15:15.372260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.372138    4515 scope.go:122] "RemoveContainer" containerID="6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"
Nov 02 08:15:15.554838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.554801    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:15:15.795258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.795089    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:15:15.795258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.795218    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:15.823426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.823395    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f765b8-dockercfg-2nstz\""
Nov 02 08:15:15.823426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.823396    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f765b8\""
Nov 02 08:15:15.955245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.955171    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75j5f\" (UniqueName: \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:15.955396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:15.955369    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.057082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.057042    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75j5f\" (UniqueName: \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.057265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.057180    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.057928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.057906    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.080904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.080872    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75j5f\" (UniqueName: \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\") pod \"payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.112852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.112813    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.366216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.366183    4515 scope.go:122] "RemoveContainer" containerID="363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"
Nov 02 08:15:16.366403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.366278    4515 scope.go:122] "RemoveContainer" containerID="6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"
Nov 02 08:15:16.427042 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:16.427013    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6072711d_30c3_48a8_b9d1_53c17aebe6a4.slice/crio-2b018d11089cd2db6b1c41f97426feafc7fa6505b62087a51d434a67372fb761 WatchSource:0}: Error finding container 2b018d11089cd2db6b1c41f97426feafc7fa6505b62087a51d434a67372fb761: Status 404 returned error can't find the container with id 2b018d11089cd2db6b1c41f97426feafc7fa6505b62087a51d434a67372fb761
Nov 02 08:15:16.429345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.429287    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:15:16.802322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.802275    4515 generic.go:356] "Generic (PLEG): container finished" podID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerID="397a066e2e0f701196a2216d7be026c77672d0f43ff4868f4e7aa490fa8f2e36" exitCode=60
Nov 02 08:15:16.802525 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.802350    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" event={"ID":"03855a29-10d7-4364-a77c-7c15dbea89ed","Type":"ContainerDied","Data":"397a066e2e0f701196a2216d7be026c77672d0f43ff4868f4e7aa490fa8f2e36"}
Nov 02 08:15:16.802525 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.802409    4515 scope.go:122] "RemoveContainer" containerID="cd4e2be7a7d441c1a9f89952588cfa6350c4d88e0ac7126b0ead417b632a02f0"
Nov 02 08:15:16.802908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.802887    4515 scope.go:122] "RemoveContainer" containerID="397a066e2e0f701196a2216d7be026c77672d0f43ff4868f4e7aa490fa8f2e36"
Nov 02 08:15:16.803331 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:16.803288    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp_odh-ai-gateway-infra(03855a29-10d7-4364-a77c-7c15dbea89ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed"
Nov 02 08:15:16.804335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.804298    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" event={"ID":"6072711d-30c3-48a8-b9d1-53c17aebe6a4","Type":"ContainerStarted","Data":"30b247a145f3d5ad2b8bafcb2ed84db6cb70d5e274070096c2bb75a69f1ec1d1"}
Nov 02 08:15:16.804442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.804366    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:16.804442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.804390    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" event={"ID":"6072711d-30c3-48a8-b9d1-53c17aebe6a4","Type":"ContainerStarted","Data":"2b018d11089cd2db6b1c41f97426feafc7fa6505b62087a51d434a67372fb761"}
Nov 02 08:15:16.806450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.806430    4515 generic.go:356] "Generic (PLEG): container finished" podID="3036233b-9580-4919-a9b7-151df8375612" containerID="6b9236eed276058debbf321bf43bd776e97015151ca0070b53bd912fd966c8a6" exitCode=60
Nov 02 08:15:16.806554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.806498    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerDied","Data":"6b9236eed276058debbf321bf43bd776e97015151ca0070b53bd912fd966c8a6"}
Nov 02 08:15:16.806904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.806883    4515 scope.go:122] "RemoveContainer" containerID="6b9236eed276058debbf321bf43bd776e97015151ca0070b53bd912fd966c8a6"
Nov 02 08:15:16.807245 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:16.807222    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx_odh-ai-gateway-infra(3036233b-9580-4919-a9b7-151df8375612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" podUID="3036233b-9580-4919-a9b7-151df8375612"
Nov 02 08:15:16.808485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.808465    4515 generic.go:356] "Generic (PLEG): container finished" podID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerID="aae71305e746ed27a2c59e2e1fafb4dcdb0635a1ebba8ea0cf3bac3cff9d360f" exitCode=60
Nov 02 08:15:16.808557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.808517    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerDied","Data":"aae71305e746ed27a2c59e2e1fafb4dcdb0635a1ebba8ea0cf3bac3cff9d360f"}
Nov 02 08:15:16.808859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.808844    4515 scope.go:122] "RemoveContainer" containerID="aae71305e746ed27a2c59e2e1fafb4dcdb0635a1ebba8ea0cf3bac3cff9d360f"
Nov 02 08:15:16.809152 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:16.809136    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq_odh-ai-gateway-infra(c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc"
Nov 02 08:15:16.819512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.819485    4515 scope.go:122] "RemoveContainer" containerID="6c815b2c41e718b3a913af126b05bcd51f1478d39319b1ef0aa74a74827d9d51"
Nov 02 08:15:16.834863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.834841    4515 scope.go:122] "RemoveContainer" containerID="b538ae0bd6aaf1615cc57e881084a305eb7319133e630af25067c4128b38acae"
Nov 02 08:15:16.974010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:16.973889    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" podStartSLOduration=1.973872592 podStartE2EDuration="1.973872592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:16.969217492 +0000 UTC m=+3061.731654397" watchObservedRunningTime="2026-11-02 08:15:16.973872592 +0000 UTC m=+3061.736309500"
Nov 02 08:15:17.378086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.378058    4515 scope.go:122] "RemoveContainer" containerID="83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"
Nov 02 08:15:17.378272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.378185    4515 scope.go:122] "RemoveContainer" containerID="7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"
Nov 02 08:15:17.726330 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:17.726284    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:15:17.821977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.821945    4515 generic.go:356] "Generic (PLEG): container finished" podID="f07106ac-f7a0-4029-b901-58387cb85744" containerID="548eb91f49aa9e208e8816df1c001b0f18a52b656c40005561e00d6797c549d2" exitCode=60
Nov 02 08:15:17.822186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.822017    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerDied","Data":"548eb91f49aa9e208e8816df1c001b0f18a52b656c40005561e00d6797c549d2"}
Nov 02 08:15:17.822186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.822095    4515 scope.go:122] "RemoveContainer" containerID="6f8d95313138efbb351387a77f44626596ae281f7c7648c0b2a9efed2b953bb8"
Nov 02 08:15:17.822633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.822608    4515 scope.go:122] "RemoveContainer" containerID="548eb91f49aa9e208e8816df1c001b0f18a52b656c40005561e00d6797c549d2"
Nov 02 08:15:17.823767 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:17.823735    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg_odh-ai-gateway-infra(f07106ac-f7a0-4029-b901-58387cb85744)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" podUID="f07106ac-f7a0-4029-b901-58387cb85744"
Nov 02 08:15:17.827380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.827353    4515 generic.go:356] "Generic (PLEG): container finished" podID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerID="fbf06dc1744294fd08bae5953545c7a11776c86d0a7eded3f09bd760e225fe6a" exitCode=60
Nov 02 08:15:17.827502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.827428    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerDied","Data":"fbf06dc1744294fd08bae5953545c7a11776c86d0a7eded3f09bd760e225fe6a"}
Nov 02 08:15:17.827963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.827941    4515 scope.go:122] "RemoveContainer" containerID="fbf06dc1744294fd08bae5953545c7a11776c86d0a7eded3f09bd760e225fe6a"
Nov 02 08:15:17.828389 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:17.828364    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6_odh-ai-gateway-infra(1b51c3bf-8bbe-4380-b2e3-c63e6555aae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2"
Nov 02 08:15:17.837563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.837543    4515 scope.go:122] "RemoveContainer" containerID="363f113e1a882b124449094c1436e7b95205bdcf3f6babbe974639fd6511db89"
Nov 02 08:15:17.856855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.856827    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:17.899822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.899719    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:17.943000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:17.940921    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:18.071209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.071175    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:18.192196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.173417    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:18.208356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.203087    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\") pod \"03855a29-10d7-4364-a77c-7c15dbea89ed\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") "
Nov 02 08:15:18.208356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.203396    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\" (UniqueName: \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\") pod \"03855a29-10d7-4364-a77c-7c15dbea89ed\" (UID: \"03855a29-10d7-4364-a77c-7c15dbea89ed\") "
Nov 02 08:15:18.208356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.205182    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca" pod "03855a29-10d7-4364-a77c-7c15dbea89ed" (UID: "03855a29-10d7-4364-a77c-7c15dbea89ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:18.208767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.208735    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4" pod "03855a29-10d7-4364-a77c-7c15dbea89ed" (UID: "03855a29-10d7-4364-a77c-7c15dbea89ed"). InnerVolumeSpecName "kube-api-access-wgcj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:18.248519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.248489    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:18.305618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.305576    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\" (UniqueName: \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\") pod \"3036233b-9580-4919-a9b7-151df8375612\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") "
Nov 02 08:15:18.305817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.305769    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\") pod \"3036233b-9580-4919-a9b7-151df8375612\" (UID: \"3036233b-9580-4919-a9b7-151df8375612\") "
Nov 02 08:15:18.306264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.306228    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca" pod "3036233b-9580-4919-a9b7-151df8375612" (UID: "3036233b-9580-4919-a9b7-151df8375612"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:18.306608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.306581    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgcj4\" (UniqueName: \"kubernetes.io/projected/03855a29-10d7-4364-a77c-7c15dbea89ed-kube-api-access-wgcj4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.306701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.306613    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03855a29-10d7-4364-a77c-7c15dbea89ed-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.306701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.306628    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3036233b-9580-4919-a9b7-151df8375612-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.308043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.308017    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc" pod "3036233b-9580-4919-a9b7-151df8375612" (UID: "3036233b-9580-4919-a9b7-151df8375612"). InnerVolumeSpecName "kube-api-access-mw5dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:18.366673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.366642    4515 scope.go:122] "RemoveContainer" containerID="63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"
Nov 02 08:15:18.407612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.407584    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\" (UniqueName: \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\") pod \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") "
Nov 02 08:15:18.407751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.407741    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\") pod \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\" (UID: \"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc\") "
Nov 02 08:15:18.408201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.408172    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca" pod "c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" (UID: "c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:18.408596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.408571    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.408687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.408601    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5dc\" (UniqueName: \"kubernetes.io/projected/3036233b-9580-4919-a9b7-151df8375612-kube-api-access-mw5dc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.409825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.409803    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m" pod "c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" (UID: "c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc"). InnerVolumeSpecName "kube-api-access-8xj5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:18.510141 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.510069    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj5m\" (UniqueName: \"kubernetes.io/projected/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc-kube-api-access-8xj5m\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:18.841206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.841126    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"
Nov 02 08:15:18.841206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.841163    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp" event={"ID":"03855a29-10d7-4364-a77c-7c15dbea89ed","Type":"ContainerDied","Data":"b3e527255023f0ef99d4b3e316983f8ec940cfe5f8fd6ecacc6c1a07b03e0cbd"}
Nov 02 08:15:18.841747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.841231    4515 scope.go:122] "RemoveContainer" containerID="397a066e2e0f701196a2216d7be026c77672d0f43ff4868f4e7aa490fa8f2e36"
Nov 02 08:15:18.846343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.846302    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerStarted","Data":"de9610d3ea3ca1fb0eee43e405a50d2f7ce3e83a74ce60aa17c0576abcb1ed18"}
Nov 02 08:15:18.849304 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.849278    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"
Nov 02 08:15:18.849304 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.849318    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx" event={"ID":"3036233b-9580-4919-a9b7-151df8375612","Type":"ContainerDied","Data":"2688f22e24cae6a486f47761bdea409ae4f22dcdd3e66df459861da625ec07e6"}
Nov 02 08:15:18.853159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.853141    4515 scope.go:122] "RemoveContainer" containerID="6b9236eed276058debbf321bf43bd776e97015151ca0070b53bd912fd966c8a6"
Nov 02 08:15:18.853352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.853325    4515 generic.go:356] "Generic (PLEG): container finished" podID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerID="fb58ce4c0a716bd5539248142963796ceb831f0d5b95956ad6c3737ad5a1f660" exitCode=60
Nov 02 08:15:18.853446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.853332    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerDied","Data":"fb58ce4c0a716bd5539248142963796ceb831f0d5b95956ad6c3737ad5a1f660"}
Nov 02 08:15:18.853824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.853807    4515 scope.go:122] "RemoveContainer" containerID="fb58ce4c0a716bd5539248142963796ceb831f0d5b95956ad6c3737ad5a1f660"
Nov 02 08:15:18.854127 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:18.854098    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr_odh-ai-gateway-infra(828dc10b-98ae-4ea9-95fc-b34a6fd6447c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c"
Nov 02 08:15:18.855594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.855571    4515 generic.go:356] "Generic (PLEG): container finished" podID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerID="6f6af4e3cf1d5caddf35a3c8669f317bbd014816dd1cecdfe4ea723431ce61aa" exitCode=60
Nov 02 08:15:18.855681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.855638    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerDied","Data":"6f6af4e3cf1d5caddf35a3c8669f317bbd014816dd1cecdfe4ea723431ce61aa"}
Nov 02 08:15:18.856073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.856051    4515 scope.go:122] "RemoveContainer" containerID="6f6af4e3cf1d5caddf35a3c8669f317bbd014816dd1cecdfe4ea723431ce61aa"
Nov 02 08:15:18.857214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.857196    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"
Nov 02 08:15:18.857334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.857224    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq" event={"ID":"c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc","Type":"ContainerDied","Data":"ec4df5110eea4cab10abb659a238c5f6a8038d97e262733c16ce4e404895b8f4"}
Nov 02 08:15:18.857548 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:18.857518    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz_odh-ai-gateway-infra(46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d"
Nov 02 08:15:18.864444 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.864424    4515 scope.go:122] "RemoveContainer" containerID="83899780941b6e26e1e2caa2239deaf2b8a3afebab377288f8bdfb8ffe094760"
Nov 02 08:15:18.877914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.877887    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:18.879350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.879331    4515 scope.go:122] "RemoveContainer" containerID="7765815b972105bddce76d5b52e80e4aaff5a7a3282b2de2660d92eea24759c9"
Nov 02 08:15:18.897379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.897357    4515 scope.go:122] "RemoveContainer" containerID="aae71305e746ed27a2c59e2e1fafb4dcdb0635a1ebba8ea0cf3bac3cff9d360f"
Nov 02 08:15:18.940142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.940105    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:18.947099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.947070    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893455-2f4vp"]
Nov 02 08:15:18.969919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:18.969686    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:19.075591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.075562    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:19.092135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.092051    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:19.102603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.102571    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893455-zlqlq"]
Nov 02 08:15:19.149951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.149926    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:19.171606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.171577    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:19.177056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.177028    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893455-gg6vx"]
Nov 02 08:15:19.221190 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.221157    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\" (UniqueName: \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\") pod \"f07106ac-f7a0-4029-b901-58387cb85744\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") "
Nov 02 08:15:19.221402 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.221269    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\") pod \"f07106ac-f7a0-4029-b901-58387cb85744\" (UID: \"f07106ac-f7a0-4029-b901-58387cb85744\") "
Nov 02 08:15:19.221614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.221592    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca" pod "f07106ac-f7a0-4029-b901-58387cb85744" (UID: "f07106ac-f7a0-4029-b901-58387cb85744"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:19.221899 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.221879    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07106ac-f7a0-4029-b901-58387cb85744-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:19.223651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.223629    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw" pod "f07106ac-f7a0-4029-b901-58387cb85744" (UID: "f07106ac-f7a0-4029-b901-58387cb85744"). InnerVolumeSpecName "kube-api-access-dxbtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:19.323192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.323160    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\") pod \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") "
Nov 02 08:15:19.323468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.323447    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\" (UniqueName: \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\") pod \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\" (UID: \"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2\") "
Nov 02 08:15:19.323738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.323710    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca" pod "1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" (UID: "1b51c3bf-8bbe-4380-b2e3-c63e6555aae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:19.324143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.324121    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbtw\" (UniqueName: \"kubernetes.io/projected/f07106ac-f7a0-4029-b901-58387cb85744-kube-api-access-dxbtw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:19.324266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.324150    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:19.325864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.325840    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr" pod "1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" (UID: "1b51c3bf-8bbe-4380-b2e3-c63e6555aae2"). InnerVolumeSpecName "kube-api-access-n6qkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:19.366788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.366721    4515 scope.go:122] "RemoveContainer" containerID="99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"
Nov 02 08:15:19.384871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.382744    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" path="/var/lib/kubelet/pods/03855a29-10d7-4364-a77c-7c15dbea89ed/volumes"
Nov 02 08:15:19.384871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.383554    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3036233b-9580-4919-a9b7-151df8375612" path="/var/lib/kubelet/pods/3036233b-9580-4919-a9b7-151df8375612/volumes"
Nov 02 08:15:19.384871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.384233    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" path="/var/lib/kubelet/pods/c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc/volumes"
Nov 02 08:15:19.425363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.425337    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6qkr\" (UniqueName: \"kubernetes.io/projected/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2-kube-api-access-n6qkr\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:19.868118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.868081    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg" event={"ID":"f07106ac-f7a0-4029-b901-58387cb85744","Type":"ContainerDied","Data":"80c227a307ddc176805d8be1ddc986e2189b42a072fcdc5e82b3782d0705ee22"}
Nov 02 08:15:19.868595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.868146    4515 scope.go:122] "RemoveContainer" containerID="548eb91f49aa9e208e8816df1c001b0f18a52b656c40005561e00d6797c549d2"
Nov 02 08:15:19.868595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.868091    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"
Nov 02 08:15:19.870369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.870341    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerStarted","Data":"f5df0438fab27c7267e5c080a6f4c8e119d19d594ef4576e114a2fb6872e0366"}
Nov 02 08:15:19.873543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.873523    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"
Nov 02 08:15:19.873543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.873532    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6" event={"ID":"1b51c3bf-8bbe-4380-b2e3-c63e6555aae2","Type":"ContainerDied","Data":"c9a1610906454b6907491d5010878230d695d6167dcac1709185c4f1f462cbf9"}
Nov 02 08:15:19.875699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.875673    4515 generic.go:356] "Generic (PLEG): container finished" podID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerID="de9610d3ea3ca1fb0eee43e405a50d2f7ce3e83a74ce60aa17c0576abcb1ed18" exitCode=60
Nov 02 08:15:19.875800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.875700    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerDied","Data":"de9610d3ea3ca1fb0eee43e405a50d2f7ce3e83a74ce60aa17c0576abcb1ed18"}
Nov 02 08:15:19.876168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.876147    4515 scope.go:122] "RemoveContainer" containerID="de9610d3ea3ca1fb0eee43e405a50d2f7ce3e83a74ce60aa17c0576abcb1ed18"
Nov 02 08:15:19.876533 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:19.876506    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw_odh-ai-gateway-infra(4d4857be-3490-4bdc-8dee-7706fee266cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb"
Nov 02 08:15:19.881961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.881945    4515 scope.go:122] "RemoveContainer" containerID="fbf06dc1744294fd08bae5953545c7a11776c86d0a7eded3f09bd760e225fe6a"
Nov 02 08:15:19.892930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.892909    4515 scope.go:122] "RemoveContainer" containerID="63ed8b842f5b517125f69320f8fcacbe2e3685ce08fc70321f88c9a77b931607"
Nov 02 08:15:19.910802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.910778    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:19.937869 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.937815    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:19.981395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:19.981295    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4489d-29893455-p9lrg"]
Nov 02 08:15:20.003761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.003728    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:20.030505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.029504    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:20.038617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.038541    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893455-c2bb6"]
Nov 02 08:15:20.061038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.061004    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:20.191616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.191586    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:20.304883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.304854    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:20.353481 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.353443    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\") pod \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") "
Nov 02 08:15:20.353649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.353609    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\" (UniqueName: \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\") pod \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\" (UID: \"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d\") "
Nov 02 08:15:20.354013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.353984    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca" pod "46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" (UID: "46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.354438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.354415    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.356983 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.356948    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg" pod "46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" (UID: "46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d"). InnerVolumeSpecName "kube-api-access-6qzgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.455728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.455699    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\") pod \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") "
Nov 02 08:15:20.455897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.455881    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\") pod \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\" (UID: \"828dc10b-98ae-4ea9-95fc-b34a6fd6447c\") "
Nov 02 08:15:20.456187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.456156    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca" pod "828dc10b-98ae-4ea9-95fc-b34a6fd6447c" (UID: "828dc10b-98ae-4ea9-95fc-b34a6fd6447c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.456703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.456679    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzgg\" (UniqueName: \"kubernetes.io/projected/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d-kube-api-access-6qzgg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.456835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.456707    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.458532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.458502    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n" pod "828dc10b-98ae-4ea9-95fc-b34a6fd6447c" (UID: "828dc10b-98ae-4ea9-95fc-b34a6fd6447c"). InnerVolumeSpecName "kube-api-access-rbw6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.558747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.558654    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/828dc10b-98ae-4ea9-95fc-b34a6fd6447c-kube-api-access-rbw6n\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.892199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.892117    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"
Nov 02 08:15:20.892199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.892128    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr" event={"ID":"828dc10b-98ae-4ea9-95fc-b34a6fd6447c","Type":"ContainerDied","Data":"a11d7cdbe0b2025d0b553241bc11e00f9970d20b126e2a74b67b2ff720994b34"}
Nov 02 08:15:20.892887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.892197    4515 scope.go:122] "RemoveContainer" containerID="fb58ce4c0a716bd5539248142963796ceb831f0d5b95956ad6c3737ad5a1f660"
Nov 02 08:15:20.894555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.894532    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"
Nov 02 08:15:20.894663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.894557    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz" event={"ID":"46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d","Type":"ContainerDied","Data":"8a1bb620f06b5c370e6764ed2a0d1d762deb7e733f7d3f68e22b2b332f0ce1d6"}
Nov 02 08:15:20.902890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.902868    4515 generic.go:356] "Generic (PLEG): container finished" podID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerID="f5df0438fab27c7267e5c080a6f4c8e119d19d594ef4576e114a2fb6872e0366" exitCode=60
Nov 02 08:15:20.903012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.902939    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerDied","Data":"f5df0438fab27c7267e5c080a6f4c8e119d19d594ef4576e114a2fb6872e0366"}
Nov 02 08:15:20.903715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.903461    4515 scope.go:122] "RemoveContainer" containerID="f5df0438fab27c7267e5c080a6f4c8e119d19d594ef4576e114a2fb6872e0366"
Nov 02 08:15:20.903969 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:20.903945    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893455-6dp2g_odh-ai-gateway-infra(707314c1-4961-42b4-94c0-dad02c5ad2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe"
Nov 02 08:15:20.905007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.904988    4515 scope.go:122] "RemoveContainer" containerID="6f6af4e3cf1d5caddf35a3c8669f317bbd014816dd1cecdfe4ea723431ce61aa"
Nov 02 08:15:20.918504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:20.918477    4515 scope.go:122] "RemoveContainer" containerID="99a3aa00b0f47e065ac5f714bfc29f56d4de2bb01a1aca6763eb71a264f98d3b"
Nov 02 08:15:21.059129 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.059096    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:21.069089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.069061    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893455-9ffrr"]
Nov 02 08:15:21.083152 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.083125    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:21.089568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.089538    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:21.114084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.114053    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:21.120818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.120788    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893455-n5hzz"]
Nov 02 08:15:21.268505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.268471    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\") pod \"4d4857be-3490-4bdc-8dee-7706fee266cb\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") "
Nov 02 08:15:21.268677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.268653    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\" (UniqueName: \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\") pod \"4d4857be-3490-4bdc-8dee-7706fee266cb\" (UID: \"4d4857be-3490-4bdc-8dee-7706fee266cb\") "
Nov 02 08:15:21.268891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.268863    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca" pod "4d4857be-3490-4bdc-8dee-7706fee266cb" (UID: "4d4857be-3490-4bdc-8dee-7706fee266cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:21.269209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.269186    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4857be-3490-4bdc-8dee-7706fee266cb-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:21.270928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.270899    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph" pod "4d4857be-3490-4bdc-8dee-7706fee266cb" (UID: "4d4857be-3490-4bdc-8dee-7706fee266cb"). InnerVolumeSpecName "kube-api-access-vs6ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:21.370058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.370025    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6ph\" (UniqueName: \"kubernetes.io/projected/4d4857be-3490-4bdc-8dee-7706fee266cb-kube-api-access-vs6ph\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:21.376294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.376264    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" path="/var/lib/kubelet/pods/1b51c3bf-8bbe-4380-b2e3-c63e6555aae2/volumes"
Nov 02 08:15:21.376744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.376729    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" path="/var/lib/kubelet/pods/46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d/volumes"
Nov 02 08:15:21.377148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.377134    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" path="/var/lib/kubelet/pods/828dc10b-98ae-4ea9-95fc-b34a6fd6447c/volumes"
Nov 02 08:15:21.377928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.377912    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07106ac-f7a0-4029-b901-58387cb85744" path="/var/lib/kubelet/pods/f07106ac-f7a0-4029-b901-58387cb85744/volumes"
Nov 02 08:15:21.915107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.915074    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"
Nov 02 08:15:21.915107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.915091    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw" event={"ID":"4d4857be-3490-4bdc-8dee-7706fee266cb","Type":"ContainerDied","Data":"fc07b1a93a8e2a8d6eaa35adabef5e9f46034b6867882b46829cc20382fc9eec"}
Nov 02 08:15:21.915648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.915158    4515 scope.go:122] "RemoveContainer" containerID="de9610d3ea3ca1fb0eee43e405a50d2f7ce3e83a74ce60aa17c0576abcb1ed18"
Nov 02 08:15:21.971490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.971457    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:21.983345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:21.983299    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893455-sx9jw"]
Nov 02 08:15:22.160093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.160067    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:22.193459 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.193390    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\") pod \"707314c1-4961-42b4-94c0-dad02c5ad2fe\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") "
Nov 02 08:15:22.193610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.193511    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\" (UniqueName: \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\") pod \"707314c1-4961-42b4-94c0-dad02c5ad2fe\" (UID: \"707314c1-4961-42b4-94c0-dad02c5ad2fe\") "
Nov 02 08:15:22.193772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.193749    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca" pod "707314c1-4961-42b4-94c0-dad02c5ad2fe" (UID: "707314c1-4961-42b4-94c0-dad02c5ad2fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:22.194089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.194061    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707314c1-4961-42b4-94c0-dad02c5ad2fe-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.195699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.195672    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff" pod "707314c1-4961-42b4-94c0-dad02c5ad2fe" (UID: "707314c1-4961-42b4-94c0-dad02c5ad2fe"). InnerVolumeSpecName "kube-api-access-jjjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:22.295266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.295230    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjff\" (UniqueName: \"kubernetes.io/projected/707314c1-4961-42b4-94c0-dad02c5ad2fe-kube-api-access-jjjff\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.930669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.930635    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"
Nov 02 08:15:22.931194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.930669    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g" event={"ID":"707314c1-4961-42b4-94c0-dad02c5ad2fe","Type":"ContainerDied","Data":"234f75c80a45936d6089bc8668b712cd94a1b53edd92524bc235f54c03710300"}
Nov 02 08:15:22.931194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.930731    4515 scope.go:122] "RemoveContainer" containerID="f5df0438fab27c7267e5c080a6f4c8e119d19d594ef4576e114a2fb6872e0366"
Nov 02 08:15:22.993671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:22.993638    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:23.004507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:23.004467    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893455-6dp2g"]
Nov 02 08:15:23.387130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:23.387086    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" path="/var/lib/kubelet/pods/4d4857be-3490-4bdc-8dee-7706fee266cb/volumes"
Nov 02 08:15:23.388041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:23.388013    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" path="/var/lib/kubelet/pods/707314c1-4961-42b4-94c0-dad02c5ad2fe/volumes"
Nov 02 08:15:27.835280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:27.835248    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:15:46.127646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:46.127610    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:15:47.729345 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:47.729270    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:15:51.128292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.128248    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" podUID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" containerName="payload-processing" containerID="cri-o://2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21" gracePeriod=25
Nov 02 08:15:51.429664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.429636    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:15:51.442238 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442207    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\") pod \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") "
Nov 02 08:15:51.442571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442547    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\") pod \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") "
Nov 02 08:15:51.442663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442610    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\") pod \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") "
Nov 02 08:15:51.442729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442696    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\") pod \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\" (UID: \"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98\") "
Nov 02 08:15:51.442729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442716    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume" pod "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" (UID: "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:51.442833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.442749    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume" pod "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" (UID: "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:51.443384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.443360    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:51.443490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.443388    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:51.444958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.444930    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials" pod "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" (UID: "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:51.445188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.445163    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9" pod "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" (UID: "3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98"). InnerVolumeSpecName "kube-api-access-bg8c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:51.587333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.552897    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8c9\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-kube-api-access-bg8c9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:51.587333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:51.552940    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:15:52.216886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.216833    4515 generic.go:356] "Generic (PLEG): container finished" podID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" containerID="2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21" exitCode=0
Nov 02 08:15:52.217796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.217763    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" event={"ID":"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98","Type":"ContainerDied","Data":"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"}
Nov 02 08:15:52.218002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.217984    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z" event={"ID":"3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98","Type":"ContainerDied","Data":"40a8d505a1c4b03842238f77fcf6d3153ec12d146dff8079d3e7f4bbec248419"}
Nov 02 08:15:52.218121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.218108    4515 scope.go:122] "RemoveContainer" containerID="2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"
Nov 02 08:15:52.218404 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.218385    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"
Nov 02 08:15:52.273349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.272132    4515 scope.go:122] "RemoveContainer" containerID="2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"
Nov 02 08:15:52.273349 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:15:52.272594    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21\": container with ID starting with 2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21 not found: ID does not exist" containerID="2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"
Nov 02 08:15:52.273349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.272637    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21"} err="failed to get container status \"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21\": rpc error: code = NotFound desc = could not find container \"2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21\": container with ID starting with 2d8a227fa5e26a9df49f65ea9b866b85d7fa6e37161b9366685f476a5950dd21 not found: ID does not exist"
Nov 02 08:15:52.312818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.312774    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:15:52.321271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:52.321229    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5878vz72z"]
Nov 02 08:15:53.375225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.375190    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" path="/var/lib/kubelet/pods/3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98/volumes"
Nov 02 08:15:53.768511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.768475    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:15:53.769519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769497    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.769519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769519    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769532    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769537    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769547    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769552    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769559    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769565    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769573    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769578    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769591    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769595    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769605    4515 cpu_manager.go:414] "containerMap: removing container" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769610    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769618    4515 cpu_manager.go:414] "containerMap: removing container" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769623    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769630    4515 cpu_manager.go:414] "containerMap: removing container" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769635    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769644    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769649    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.769652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769659    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" containerName="payload-processing"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769663    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" containerName="payload-processing"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769672    4515 cpu_manager.go:414] "containerMap: removing container" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769677    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769685    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769690    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769695    4515 cpu_manager.go:414] "containerMap: removing container" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769699    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769709    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769713    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769724    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769728    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769735    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769739    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769747    4515 cpu_manager.go:414] "containerMap: removing container" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769751    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769762    4515 cpu_manager.go:414] "containerMap: removing container" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769767    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769774    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769779    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769790    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769794    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769807    4515 cpu_manager.go:414] "containerMap: removing container" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769811    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769819    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769823    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769919    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769930    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769936    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769941    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769946    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769953    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5d7b35-81e4-45a9-b37d-fc45ba8e5b98" containerName="payload-processing"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769958    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769963    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769968    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769974    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769979    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769984    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769989    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.769995    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770001    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770006    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770012    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770017    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770023    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.770291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.770030    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.775156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.775136    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:53.795215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.795177    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:15:53.801396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.801361    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dfd6f0-dockercfg-kxhqm\""
Nov 02 08:15:53.801396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.801361    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dfd6f0\""
Nov 02 08:15:53.912920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.912878    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrk4\" (UniqueName: \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:53.912920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.912927    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:53.925055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.925017    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:15:53.926559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.926530    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.926767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.926747    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.926898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.926883    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.927002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.926989    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.927155 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927139    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.927245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927233    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.927384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927371    4515 cpu_manager.go:414] "containerMap: removing container" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.927467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927456    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.927609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927596    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.927689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927678    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b51c3bf-8bbe-4380-b2e3-c63e6555aae2" containerName="cleanup"
Nov 02 08:15:53.927984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.927964    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="707314c1-4961-42b4-94c0-dad02c5ad2fe" containerName="cleanup"
Nov 02 08:15:53.928108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928096    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03855a29-10d7-4364-a77c-7c15dbea89ed" containerName="cleanup"
Nov 02 08:15:53.928213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928201    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07106ac-f7a0-4029-b901-58387cb85744" containerName="cleanup"
Nov 02 08:15:53.928320 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928293    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="828dc10b-98ae-4ea9-95fc-b34a6fd6447c" containerName="cleanup"
Nov 02 08:15:53.928423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928410    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a653e6-e7e0-4fd1-8b0e-7d46d09ef88d" containerName="cleanup"
Nov 02 08:15:53.928529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928516    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fd1c09-42ca-4dc9-ac24-c3d8fb023bcc" containerName="cleanup"
Nov 02 08:15:53.928632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928619    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4857be-3490-4bdc-8dee-7706fee266cb" containerName="cleanup"
Nov 02 08:15:53.928723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.928711    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3036233b-9580-4919-a9b7-151df8375612" containerName="cleanup"
Nov 02 08:15:53.951755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:53.942504    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.003776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.003742    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:15:54.037387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.020959    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.037387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.021029    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjlz\" (UniqueName: \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.037387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.021073    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrk4\" (UniqueName: \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:54.037387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.021139    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:54.037387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.022293    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:54.126071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.126025    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.127489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.126326    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjlz\" (UniqueName: \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.128052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.128029    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.134865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.133766    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrk4\" (UniqueName: \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\") pod \"payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:54.165342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.165214    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjlz\" (UniqueName: \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\") pod \"payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.304497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.304407    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:54.394976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.394936    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:54.616177 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:54.616143    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55681672_5be5_4164_8d46_3be78e39c8ba.slice/crio-01a8027616ff68a8a5f09c8a6b6ee590b2e8158c888ff99590de8d968cdc97c8 WatchSource:0}: Error finding container 01a8027616ff68a8a5f09c8a6b6ee590b2e8158c888ff99590de8d968cdc97c8: Status 404 returned error can't find the container with id 01a8027616ff68a8a5f09c8a6b6ee590b2e8158c888ff99590de8d968cdc97c8
Nov 02 08:15:54.618781 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.618754    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:15:54.709841 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:15:54.709813    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1094dfa4_e1c2_4d0c_814f_0ea0825a704d.slice/crio-529514075d23c5b79e8ecd7ac57667a9e6439ab386df04d7f5fbd04a0d59bd88 WatchSource:0}: Error finding container 529514075d23c5b79e8ecd7ac57667a9e6439ab386df04d7f5fbd04a0d59bd88: Status 404 returned error can't find the container with id 529514075d23c5b79e8ecd7ac57667a9e6439ab386df04d7f5fbd04a0d59bd88
Nov 02 08:15:54.712174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:54.712151    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:15:55.244417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.244373    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" event={"ID":"1094dfa4-e1c2-4d0c-814f-0ea0825a704d","Type":"ContainerStarted","Data":"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"}
Nov 02 08:15:55.244639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.244446    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" event={"ID":"1094dfa4-e1c2-4d0c-814f-0ea0825a704d","Type":"ContainerStarted","Data":"529514075d23c5b79e8ecd7ac57667a9e6439ab386df04d7f5fbd04a0d59bd88"}
Nov 02 08:15:55.244639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.244477    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:15:55.246319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.246274    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" event={"ID":"55681672-5be5-4164-8d46-3be78e39c8ba","Type":"ContainerStarted","Data":"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"}
Nov 02 08:15:55.246468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.246360    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" event={"ID":"55681672-5be5-4164-8d46-3be78e39c8ba","Type":"ContainerStarted","Data":"01a8027616ff68a8a5f09c8a6b6ee590b2e8158c888ff99590de8d968cdc97c8"}
Nov 02 08:15:55.246468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.246384    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:15:55.286249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.286193    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" podStartSLOduration=2.28618135 podStartE2EDuration="2.28618135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:55.28557107 +0000 UTC m=+3100.048007977" watchObservedRunningTime="2026-11-02 08:15:55.28618135 +0000 UTC m=+3100.048618253"
Nov 02 08:15:55.342700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:55.342638    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" podStartSLOduration=2.342623244 podStartE2EDuration="2.342623244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:55.326481881 +0000 UTC m=+3100.088918790" watchObservedRunningTime="2026-11-02 08:15:55.342623244 +0000 UTC m=+3100.105060150"
Nov 02 08:15:59.743516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.743429    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:15:59.748561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.748540    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:15:59.766212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.766180    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:15:59.773818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.773792    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e6f4fe\""
Nov 02 08:15:59.774135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.774115    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e6f4fe-dockercfg-lxrhz\""
Nov 02 08:15:59.945389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.945354    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:15:59.961333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.954877    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:15:59.961333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.958608    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvnt\" (UniqueName: \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:15:59.971550 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.971518    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:15:59.971920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:15:59.971843    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.076290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.076246    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:00.076526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.076508    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.076611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.076578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvnt\" (UniqueName: \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:00.076670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.076608    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdvp\" (UniqueName: \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.078967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.078932    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:00.102046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.102005    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvnt\" (UniqueName: \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\") pod \"payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:00.177932 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.177893    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.178136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.177957    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdvp\" (UniqueName: \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.178751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.178730    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.202480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.202448    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdvp\" (UniqueName: \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\") pod \"payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.322809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.322754    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:00.365091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.365055    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:00.637377 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:00.637347    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e6cce6_585b_4c66_bcf1_ee881a59052b.slice/crio-bc2d4cee1c24d02eb742d45b2ad0caf09b091a50b03a3e5152fe78497c8e0261 WatchSource:0}: Error finding container bc2d4cee1c24d02eb742d45b2ad0caf09b091a50b03a3e5152fe78497c8e0261: Status 404 returned error can't find the container with id bc2d4cee1c24d02eb742d45b2ad0caf09b091a50b03a3e5152fe78497c8e0261
Nov 02 08:16:00.639727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.639703    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:16:00.700707 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:00.700684    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:16:01.295194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.295150    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" event={"ID":"221d40e4-336f-46c8-b6b8-466427f600fc","Type":"ContainerStarted","Data":"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"}
Nov 02 08:16:01.295735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.295213    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" event={"ID":"221d40e4-336f-46c8-b6b8-466427f600fc","Type":"ContainerStarted","Data":"426f51dde5173357fc1e7bc0ed6eb240009cbb1099c2ab66b28d5e3a35ab50a1"}
Nov 02 08:16:01.295735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.295243    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:01.296964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.296939    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" event={"ID":"f8e6cce6-585b-4c66-bcf1-ee881a59052b","Type":"ContainerStarted","Data":"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"}
Nov 02 08:16:01.297062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.296989    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" event={"ID":"f8e6cce6-585b-4c66-bcf1-ee881a59052b","Type":"ContainerStarted","Data":"bc2d4cee1c24d02eb742d45b2ad0caf09b091a50b03a3e5152fe78497c8e0261"}
Nov 02 08:16:01.297062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.297024    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:01.339327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.339262    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" podStartSLOduration=2.339245979 podStartE2EDuration="2.339245979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:01.336211237 +0000 UTC m=+3106.098648140" watchObservedRunningTime="2026-11-02 08:16:01.339245979 +0000 UTC m=+3106.101682885"
Nov 02 08:16:01.380661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:01.380605    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" podStartSLOduration=2.380590984 podStartE2EDuration="2.380590984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:01.377280889 +0000 UTC m=+3106.139717796" watchObservedRunningTime="2026-11-02 08:16:01.380590984 +0000 UTC m=+3106.143027890"
Nov 02 08:16:06.256259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:06.256229    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:16:06.256723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:06.256343    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:16:12.308459 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:12.308427    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:16:12.308873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:12.308506    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:16:17.726254 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:17.726213    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:16:21.541934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.541882    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:21.550919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.550890    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:21.551214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.551197    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.725606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.725516    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.725606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.725585    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.725815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.725724    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.725948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.725933    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkfj\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.827508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.827469    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.827696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.827530    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.827696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.827616    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.827884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.827859    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkfj\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.828274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.828246    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.828429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.828405    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.830171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.830154    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.851545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.851516    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkfj\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:21.870373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:21.870341    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:22.179569 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:22.179536    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0656eb89_074a_43c7_a5e5_e7279e3bd302.slice/crio-b32d236bf3e44bb9234f33e81570908d87d6670335a9ba1f251fd90f07bb3f6f WatchSource:0}: Error finding container b32d236bf3e44bb9234f33e81570908d87d6670335a9ba1f251fd90f07bb3f6f: Status 404 returned error can't find the container with id b32d236bf3e44bb9234f33e81570908d87d6670335a9ba1f251fd90f07bb3f6f
Nov 02 08:16:22.181493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.181472    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:22.336782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.336739    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"]
Nov 02 08:16:22.342498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.342474    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.361170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.361141    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"]
Nov 02 08:16:22.544354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.538719    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2kt\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.544354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.539041    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.544354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.539245    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.544354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.539378    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.546598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.546566    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" event={"ID":"0656eb89-074a-43c7-a5e5-e7279e3bd302","Type":"ContainerStarted","Data":"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"}
Nov 02 08:16:22.546760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.546668    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:22.546760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.546692    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" event={"ID":"0656eb89-074a-43c7-a5e5-e7279e3bd302","Type":"ContainerStarted","Data":"b32d236bf3e44bb9234f33e81570908d87d6670335a9ba1f251fd90f07bb3f6f"}
Nov 02 08:16:22.591805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.591755    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" podStartSLOduration=1.591742359 podStartE2EDuration="1.591742359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:22.590677499 +0000 UTC m=+3127.353114405" watchObservedRunningTime="2026-11-02 08:16:22.591742359 +0000 UTC m=+3127.354179265"
Nov 02 08:16:22.641235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.641188    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.641453 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641290    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696c7a/agc-cred-507d8cea5cf2: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:22.641453 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641345    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:22.641453 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641410    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:23.141389911 +0000 UTC m=+3127.903826809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:22.641453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.641295    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.641641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.641549    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.641714 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.641688    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2kt\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.641780 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641712    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-643355/agc-cred-9af54fb2dfa8: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:22.641780 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641729    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:22.641780 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:22.641774    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials podName:6e331453-0811-4deb-808e-907df1b19ba7 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:23.141757867 +0000 UTC m=+3127.904194764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct" (UID: "6e331453-0811-4deb-808e-907df1b19ba7") : secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:22.642115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.642086    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.642115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.642091    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:22.701334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:22.686099    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2kt\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:23.150391 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:23.150342    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150473    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-643355/agc-cred-9af54fb2dfa8: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150498    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150563    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials podName:6e331453-0811-4deb-808e-907df1b19ba7 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:24.150543268 +0000 UTC m=+3128.912980164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct" (UID: "6e331453-0811-4deb-808e-907df1b19ba7") : secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150608    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696c7a/agc-cred-507d8cea5cf2: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150621    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:23.150675 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:23.150680    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:24.150667015 +0000 UTC m=+3128.913103920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:24.164733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:24.164696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164858    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-643355/agc-cred-9af54fb2dfa8: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164885    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164904    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696c7a/agc-cred-507d8cea5cf2: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164916    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164971    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials podName:6e331453-0811-4deb-808e-907df1b19ba7 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:26.164952802 +0000 UTC m=+3130.927389702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct" (UID: "6e331453-0811-4deb-808e-907df1b19ba7") : secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:24.165188 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.164986    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:26.164979842 +0000 UTC m=+3130.927416726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:24.458533 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:24.458449    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:16:24.471439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:24.471409    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:24.562911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:24.562878    4515 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" secret="" err="secret \"payload-processing-external-model-e2e--4bba7e07-dockercfg-gp4vp\" not found"
Nov 02 08:16:24.673892 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.673854    4515 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-plugins-e2e-extmod-a-696c7a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:24.674094 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:24.673969    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:25.173947094 +0000 UTC m=+3129.936383990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:25.183449 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:25.183418    4515 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-plugins-e2e-extmod-a-696c7a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:25.183860 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:25.183493    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:26.183477705 +0000 UTC m=+3130.945914592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:25.455995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.455896    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:16:25.469082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.469039    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"]
Nov 02 08:16:25.469651 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:25.469622    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct" podUID="6e331453-0811-4deb-808e-907df1b19ba7"
Nov 02 08:16:25.570074 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.570039    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:25.580797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.580768    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:25.692493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.692458    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\") pod \"6e331453-0811-4deb-808e-907df1b19ba7\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") "
Nov 02 08:16:25.692750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.692732    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\") pod \"6e331453-0811-4deb-808e-907df1b19ba7\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") "
Nov 02 08:16:25.692904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.692783    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\") pod \"6e331453-0811-4deb-808e-907df1b19ba7\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") "
Nov 02 08:16:25.692972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.692880    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume" pod "6e331453-0811-4deb-808e-907df1b19ba7" (UID: "6e331453-0811-4deb-808e-907df1b19ba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:25.693161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.693124    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume" pod "6e331453-0811-4deb-808e-907df1b19ba7" (UID: "6e331453-0811-4deb-808e-907df1b19ba7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:25.693555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.693535    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:25.693555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.693556    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e331453-0811-4deb-808e-907df1b19ba7-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:25.695273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.695243    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt" pod "6e331453-0811-4deb-808e-907df1b19ba7" (UID: "6e331453-0811-4deb-808e-907df1b19ba7"). InnerVolumeSpecName "kube-api-access-wt2kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:25.820339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:25.799589    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2kt\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-kube-api-access-wt2kt\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:26.229599 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229562    4515 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-plugins-e2e-extmod-a-696c7a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:26.229638    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct\" (UID: \"6e331453-0811-4deb-808e-907df1b19ba7\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229652    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:28.229631542 +0000 UTC m=+3132.992068446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229657    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696c7a/agc-cred-507d8cea5cf2: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229713    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-643355/agc-cred-9af54fb2dfa8: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229719    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229729    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct: secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229780    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:30.229765902 +0000 UTC m=+3134.992202786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:26.230012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:26.229795    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials podName:6e331453-0811-4deb-808e-907df1b19ba7 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:30.229789446 +0000 UTC m=+3134.992226330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct" (UID: "6e331453-0811-4deb-808e-907df1b19ba7") : secret "agc-cred-9af54fb2dfa8" not found
Nov 02 08:16:26.578042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:26.578009    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"
Nov 02 08:16:26.663515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:26.663471    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"]
Nov 02 08:16:26.669742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:26.669705    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-69f4sxvct"]
Nov 02 08:16:26.737654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:26.737621    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e331453-0811-4deb-808e-907df1b19ba7-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:27.444332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:27.443504    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e331453-0811-4deb-808e-907df1b19ba7" path="/var/lib/kubelet/pods/6e331453-0811-4deb-808e-907df1b19ba7/volumes"
Nov 02 08:16:27.447323 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:27.447265    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:16:28.285645 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:28.283779    4515 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-plugins-e2e-extmod-a-696c7a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:28.286103 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:28.286070    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:32.285999201 +0000 UTC m=+3137.048436105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-696c7a" not found
Nov 02 08:16:29.459377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.459338    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" podUID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" containerName="payload-processing" containerID="cri-o://feb9e1065a9203691268c92c31fcc815b75a5aa9608c4e386908220998171a48" gracePeriod=25
Nov 02 08:16:29.649089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.649018    4515 generic.go:356] "Generic (PLEG): container finished" podID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" containerID="feb9e1065a9203691268c92c31fcc815b75a5aa9608c4e386908220998171a48" exitCode=0
Nov 02 08:16:29.650148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.649720    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" event={"ID":"f25c00df-5c8b-488b-a7f7-06adaa66c16b","Type":"ContainerDied","Data":"feb9e1065a9203691268c92c31fcc815b75a5aa9608c4e386908220998171a48"}
Nov 02 08:16:29.752198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.752164    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:16:29.761186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761158    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\") pod \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") "
Nov 02 08:16:29.761359 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761266    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\") pod \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") "
Nov 02 08:16:29.761359 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761304    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\") pod \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") "
Nov 02 08:16:29.761576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761557    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\") pod \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\" (UID: \"f25c00df-5c8b-488b-a7f7-06adaa66c16b\") "
Nov 02 08:16:29.761767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761740    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume" pod "f25c00df-5c8b-488b-a7f7-06adaa66c16b" (UID: "f25c00df-5c8b-488b-a7f7-06adaa66c16b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:29.761940 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.761917    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume" pod "f25c00df-5c8b-488b-a7f7-06adaa66c16b" (UID: "f25c00df-5c8b-488b-a7f7-06adaa66c16b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:29.762285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.762265    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:29.762398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.762290    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c00df-5c8b-488b-a7f7-06adaa66c16b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:29.763536 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.763518    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials" pod "f25c00df-5c8b-488b-a7f7-06adaa66c16b" (UID: "f25c00df-5c8b-488b-a7f7-06adaa66c16b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:29.763807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.763794    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx" pod "f25c00df-5c8b-488b-a7f7-06adaa66c16b" (UID: "f25c00df-5c8b-488b-a7f7-06adaa66c16b"). InnerVolumeSpecName "kube-api-access-nrzlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:29.863545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.863512    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:29.863545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:29.863540    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrzlx\" (UniqueName: \"kubernetes.io/projected/f25c00df-5c8b-488b-a7f7-06adaa66c16b-kube-api-access-nrzlx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.268916 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:30.268883    4515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696c7a/agc-cred-507d8cea5cf2: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:30.268916 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:30.268911    4515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8: secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:30.269165 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:30.268970    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials podName:0656eb89-074a-43c7-a5e5-e7279e3bd302 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:38.26895253 +0000 UTC m=+3143.031389418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302") : secret "agc-cred-507d8cea5cf2" not found
Nov 02 08:16:30.456449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.456402    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" podUID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" containerName="payload-processing" containerID="cri-o://769d9b5a58a2d0f63cca10361b6aed86f54ba6abe20447d243338bc0a633d236" gracePeriod=25
Nov 02 08:16:30.570865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.570778    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" podUID="0656eb89-074a-43c7-a5e5-e7279e3bd302" containerName="payload-processing" containerID="cri-o://f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48" gracePeriod=25
Nov 02 08:16:30.662504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.662473    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"
Nov 02 08:16:30.662504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.662499    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn" event={"ID":"f25c00df-5c8b-488b-a7f7-06adaa66c16b","Type":"ContainerDied","Data":"74919cdc3a860cbe6deda627cee01e2c80b2f605f9ed129f2b7f764a0c94bf8c"}
Nov 02 08:16:30.662745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.662568    4515 scope.go:122] "RemoveContainer" containerID="feb9e1065a9203691268c92c31fcc815b75a5aa9608c4e386908220998171a48"
Nov 02 08:16:30.665942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.665910    4515 generic.go:356] "Generic (PLEG): container finished" podID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" containerID="769d9b5a58a2d0f63cca10361b6aed86f54ba6abe20447d243338bc0a633d236" exitCode=0
Nov 02 08:16:30.666175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.666011    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" event={"ID":"46ac439e-5a3f-48da-8ca7-98faa0b4acd8","Type":"ContainerDied","Data":"769d9b5a58a2d0f63cca10361b6aed86f54ba6abe20447d243338bc0a633d236"}
Nov 02 08:16:30.726236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.726201    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:16:30.741287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.741241    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-8f97vbrjn"]
Nov 02 08:16:30.773053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.773017    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:16:30.877273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.877238    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\") pod \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") "
Nov 02 08:16:30.877556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.877342    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\") pod \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") "
Nov 02 08:16:30.877679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.877542    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\") pod \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") "
Nov 02 08:16:30.877758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.877701    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume" pod "46ac439e-5a3f-48da-8ca7-98faa0b4acd8" (UID: "46ac439e-5a3f-48da-8ca7-98faa0b4acd8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:30.877825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.877755    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\") pod \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\" (UID: \"46ac439e-5a3f-48da-8ca7-98faa0b4acd8\") "
Nov 02 08:16:30.878411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.878387    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume" pod "46ac439e-5a3f-48da-8ca7-98faa0b4acd8" (UID: "46ac439e-5a3f-48da-8ca7-98faa0b4acd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:30.878870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.878845    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.878870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.878873    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.881964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.881936    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx" pod "46ac439e-5a3f-48da-8ca7-98faa0b4acd8" (UID: "46ac439e-5a3f-48da-8ca7-98faa0b4acd8"). InnerVolumeSpecName "kube-api-access-9mzrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:30.882082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.881982    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials" pod "46ac439e-5a3f-48da-8ca7-98faa0b4acd8" (UID: "46ac439e-5a3f-48da-8ca7-98faa0b4acd8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:30.882213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.882193    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:30.891610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.891585    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:16:30.976665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.976629    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:16:30.979442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.979413    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\") pod \"0656eb89-074a-43c7-a5e5-e7279e3bd302\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") "
Nov 02 08:16:30.980152 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.980129    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\") pod \"0656eb89-074a-43c7-a5e5-e7279e3bd302\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") "
Nov 02 08:16:30.980850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.980833    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\") pod \"0656eb89-074a-43c7-a5e5-e7279e3bd302\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") "
Nov 02 08:16:30.981072 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.980686    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume" pod "0656eb89-074a-43c7-a5e5-e7279e3bd302" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:30.981211 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.981194    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\") pod \"0656eb89-074a-43c7-a5e5-e7279e3bd302\" (UID: \"0656eb89-074a-43c7-a5e5-e7279e3bd302\") "
Nov 02 08:16:30.982441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.981703    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume" pod "0656eb89-074a-43c7-a5e5-e7279e3bd302" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982587    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj" pod "0656eb89-074a-43c7-a5e5-e7279e3bd302" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302"). InnerVolumeSpecName "kube-api-access-tlkfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982915    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzrx\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-kube-api-access-9mzrx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982934    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46ac439e-5a3f-48da-8ca7-98faa0b4acd8-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982948    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkfj\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-kube-api-access-tlkfj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982961    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.983042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.982974    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0656eb89-074a-43c7-a5e5-e7279e3bd302-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:30.985098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:30.985071    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials" pod "0656eb89-074a-43c7-a5e5-e7279e3bd302" (UID: "0656eb89-074a-43c7-a5e5-e7279e3bd302"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:31.035593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.035558    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:16:31.036881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.036844    4515 cpu_manager.go:414] "containerMap: removing container" podUID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" containerName="payload-processing"
Nov 02 08:16:31.036881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.036873    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" containerName="payload-processing"
Nov 02 08:16:31.037153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.036947    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" containerName="payload-processing"
Nov 02 08:16:31.037153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.036956    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" containerName="payload-processing"
Nov 02 08:16:31.037153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.036993    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0656eb89-074a-43c7-a5e5-e7279e3bd302" containerName="payload-processing"
Nov 02 08:16:31.037153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.037003    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0656eb89-074a-43c7-a5e5-e7279e3bd302" containerName="payload-processing"
Nov 02 08:16:31.037442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.037190    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0656eb89-074a-43c7-a5e5-e7279e3bd302" containerName="payload-processing"
Nov 02 08:16:31.037442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.037214    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" containerName="payload-processing"
Nov 02 08:16:31.037442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.037232    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" containerName="payload-processing"
Nov 02 08:16:31.047567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.047542    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.060073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.060044    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:16:31.069705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.069206    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-78ddbc-openshift-default-dockercfg-6tls9\""
Nov 02 08:16:31.084398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084368    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084415    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6vg\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084584    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084643    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084762    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084862    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084921    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084947    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.084989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.084977    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.085108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.085096    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0656eb89-074a-43c7-a5e5-e7279e3bd302-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228355    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228498    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228608    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228657    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228692    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228735    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228770    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228816    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6vg\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.229383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.228920    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.233346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.232621    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.233346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.232904    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.233643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.233617    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.237332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.234010    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.251346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.239489    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.256646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.256569    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.267223 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.267185    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6vg\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.294728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.294654    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.303339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.301011    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\") pod \"e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") " pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.368835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.368800    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:31.386198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.386149    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25c00df-5c8b-488b-a7f7-06adaa66c16b" path="/var/lib/kubelet/pods/f25c00df-5c8b-488b-a7f7-06adaa66c16b/volumes"
Nov 02 08:16:31.676577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.676539    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"
Nov 02 08:16:31.677135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.676572    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4" event={"ID":"46ac439e-5a3f-48da-8ca7-98faa0b4acd8","Type":"ContainerDied","Data":"3b15c1c2b2e11fda913c3eee6303cd66c6d438ebc89f4cbebd8689213741235c"}
Nov 02 08:16:31.677135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.676641    4515 scope.go:122] "RemoveContainer" containerID="769d9b5a58a2d0f63cca10361b6aed86f54ba6abe20447d243338bc0a633d236"
Nov 02 08:16:31.678331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.678289    4515 generic.go:356] "Generic (PLEG): container finished" podID="0656eb89-074a-43c7-a5e5-e7279e3bd302" containerID="f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48" exitCode=0
Nov 02 08:16:31.678460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.678342    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" event={"ID":"0656eb89-074a-43c7-a5e5-e7279e3bd302","Type":"ContainerDied","Data":"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"}
Nov 02 08:16:31.678460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.678363    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"
Nov 02 08:16:31.678460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.678394    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8" event={"ID":"0656eb89-074a-43c7-a5e5-e7279e3bd302","Type":"ContainerDied","Data":"b32d236bf3e44bb9234f33e81570908d87d6670335a9ba1f251fd90f07bb3f6f"}
Nov 02 08:16:31.690146 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.690126    4515 scope.go:122] "RemoveContainer" containerID="f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"
Nov 02 08:16:31.701760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.701736    4515 scope.go:122] "RemoveContainer" containerID="f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"
Nov 02 08:16:31.702122 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:31.702093    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48\": container with ID starting with f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48 not found: ID does not exist" containerID="f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"
Nov 02 08:16:31.702204 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.702137    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48"} err="failed to get container status \"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48\": rpc error: code = NotFound desc = could not find container \"f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48\": container with ID starting with f0c85b7e66d8ebe69c5bd41e785069393e986cf2158d0e36089979cadc1b0c48 not found: ID does not exist"
Nov 02 08:16:31.704483 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:31.704457    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1252f400_2f23_47b6_97c2_5e564b1725ff.slice/crio-f33282888ae63ba5aae54e051310036afd0f35aeaa5deb7bae0abc515447b661 WatchSource:0}: Error finding container f33282888ae63ba5aae54e051310036afd0f35aeaa5deb7bae0abc515447b661: Status 404 returned error can't find the container with id f33282888ae63ba5aae54e051310036afd0f35aeaa5deb7bae0abc515447b661
Nov 02 08:16:31.705023 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.704998    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:16:31.708046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.708012    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:31.708149 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.708080    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:31.708149 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.708110    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:31.750195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.750164    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:16:31.756784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.756720    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-7cbbnn2c4"]
Nov 02 08:16:31.792919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.792891    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:31.809493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:31.809460    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-9c77lwkk8"]
Nov 02 08:16:32.689230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:32.689188    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerStarted","Data":"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"}
Nov 02 08:16:32.689714 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:32.689257    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerStarted","Data":"f33282888ae63ba5aae54e051310036afd0f35aeaa5deb7bae0abc515447b661"}
Nov 02 08:16:32.747429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:32.747380    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podStartSLOduration=1.747364038 podStartE2EDuration="1.747364038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:32.746889992 +0000 UTC m=+3137.509326899" watchObservedRunningTime="2026-11-02 08:16:32.747364038 +0000 UTC m=+3137.509800944"
Nov 02 08:16:33.379703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.379666    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0656eb89-074a-43c7-a5e5-e7279e3bd302" path="/var/lib/kubelet/pods/0656eb89-074a-43c7-a5e5-e7279e3bd302/volumes"
Nov 02 08:16:33.380169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.380150    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ac439e-5a3f-48da-8ca7-98faa0b4acd8" path="/var/lib/kubelet/pods/46ac439e-5a3f-48da-8ca7-98faa0b4acd8/volumes"
Nov 02 08:16:33.380553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.380535    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:33.380656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.380618    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:33.698138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.698055    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:33.699597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:33.699574    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:16:35.892078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:35.892034    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" podUID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" containerName="payload-processing" containerID="cri-o://50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b" gracePeriod=25
Nov 02 08:16:35.977965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:35.977904    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" podUID="8788fb76-0547-49a9-90bc-c945cef42e9a" containerName="payload-processing" containerID="cri-o://1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970" gracePeriod=25
Nov 02 08:16:36.198767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.198737    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:16:36.246282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.243499    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\") pod \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") "
Nov 02 08:16:36.246282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.243637    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\") pod \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") "
Nov 02 08:16:36.246282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.243778    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\") pod \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") "
Nov 02 08:16:36.246282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.245771    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\") pod \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\" (UID: \"7e4839b5-0ab8-4fb4-b923-c81860b061d7\") "
Nov 02 08:16:36.246683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.246596    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume" pod "7e4839b5-0ab8-4fb4-b923-c81860b061d7" (UID: "7e4839b5-0ab8-4fb4-b923-c81860b061d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:36.246836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.246806    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume" pod "7e4839b5-0ab8-4fb4-b923-c81860b061d7" (UID: "7e4839b5-0ab8-4fb4-b923-c81860b061d7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:36.253532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.249811    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv" pod "7e4839b5-0ab8-4fb4-b923-c81860b061d7" (UID: "7e4839b5-0ab8-4fb4-b923-c81860b061d7"). InnerVolumeSpecName "kube-api-access-bfcqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:36.253532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.249914    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials" pod "7e4839b5-0ab8-4fb4-b923-c81860b061d7" (UID: "7e4839b5-0ab8-4fb4-b923-c81860b061d7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:36.279647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.279623    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:16:36.347104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347070    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\") pod \"8788fb76-0547-49a9-90bc-c945cef42e9a\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") "
Nov 02 08:16:36.347326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347114    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\") pod \"8788fb76-0547-49a9-90bc-c945cef42e9a\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") "
Nov 02 08:16:36.347326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347190    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\") pod \"8788fb76-0547-49a9-90bc-c945cef42e9a\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") "
Nov 02 08:16:36.347326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347272    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\") pod \"8788fb76-0547-49a9-90bc-c945cef42e9a\" (UID: \"8788fb76-0547-49a9-90bc-c945cef42e9a\") "
Nov 02 08:16:36.347636 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347604    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume" pod "8788fb76-0547-49a9-90bc-c945cef42e9a" (UID: "8788fb76-0547-49a9-90bc-c945cef42e9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:36.347636 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.347622    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume" pod "8788fb76-0547-49a9-90bc-c945cef42e9a" (UID: "8788fb76-0547-49a9-90bc-c945cef42e9a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:36.348103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348074    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.348206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348108    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.348206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348127    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8788fb76-0547-49a9-90bc-c945cef42e9a-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.348206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348141    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e4839b5-0ab8-4fb4-b923-c81860b061d7-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.348206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348155    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcqv\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-kube-api-access-bfcqv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.348206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.348172    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e4839b5-0ab8-4fb4-b923-c81860b061d7-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.350006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.349978    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk" pod "8788fb76-0547-49a9-90bc-c945cef42e9a" (UID: "8788fb76-0547-49a9-90bc-c945cef42e9a"). InnerVolumeSpecName "kube-api-access-mhbvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:36.350209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.350194    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials" pod "8788fb76-0547-49a9-90bc-c945cef42e9a" (UID: "8788fb76-0547-49a9-90bc-c945cef42e9a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:36.449390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.449279    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbvk\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-kube-api-access-mhbvk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.449390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.449330    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8788fb76-0547-49a9-90bc-c945cef42e9a-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:16:36.759369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.759289    4515 generic.go:356] "Generic (PLEG): container finished" podID="8788fb76-0547-49a9-90bc-c945cef42e9a" containerID="1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970" exitCode=0
Nov 02 08:16:36.759593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.759373    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"
Nov 02 08:16:36.759593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.759380    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" event={"ID":"8788fb76-0547-49a9-90bc-c945cef42e9a","Type":"ContainerDied","Data":"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"}
Nov 02 08:16:36.759593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.759445    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq" event={"ID":"8788fb76-0547-49a9-90bc-c945cef42e9a","Type":"ContainerDied","Data":"be3417b7876b4fd28bc108b956a858f2199ee89f990172bf4bc70feb9716bb1e"}
Nov 02 08:16:36.759593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.759476    4515 scope.go:122] "RemoveContainer" containerID="1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"
Nov 02 08:16:36.761216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.761194    4515 generic.go:356] "Generic (PLEG): container finished" podID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" containerID="50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b" exitCode=0
Nov 02 08:16:36.761337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.761239    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"
Nov 02 08:16:36.761337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.761278    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" event={"ID":"7e4839b5-0ab8-4fb4-b923-c81860b061d7","Type":"ContainerDied","Data":"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"}
Nov 02 08:16:36.761436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.761346    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26" event={"ID":"7e4839b5-0ab8-4fb4-b923-c81860b061d7","Type":"ContainerDied","Data":"ef8894dcc5aee0539f8296e990b01c678a896c47cf0ef15e29415cafd2ccd0b6"}
Nov 02 08:16:36.773505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.771649    4515 scope.go:122] "RemoveContainer" containerID="1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"
Nov 02 08:16:36.773505 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:36.772122    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970\": container with ID starting with 1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970 not found: ID does not exist" containerID="1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"
Nov 02 08:16:36.773505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.772156    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970"} err="failed to get container status \"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970\": rpc error: code = NotFound desc = could not find container \"1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970\": container with ID starting with 1d44fe182644c95ddc44fcb3f62dc7dfe8584eb3c22d6524513d4435e29e1970 not found: ID does not exist"
Nov 02 08:16:36.773505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.772179    4515 scope.go:122] "RemoveContainer" containerID="50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"
Nov 02 08:16:36.783613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.783583    4515 scope.go:122] "RemoveContainer" containerID="50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"
Nov 02 08:16:36.783971 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:36.783946    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b\": container with ID starting with 50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b not found: ID does not exist" containerID="50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"
Nov 02 08:16:36.784056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.783985    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b"} err="failed to get container status \"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b\": rpc error: code = NotFound desc = could not find container \"50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b\": container with ID starting with 50743609442f5487c8784b592c159f643a100e37fe7d15d9c24ef5ecf74e124b not found: ID does not exist"
Nov 02 08:16:36.817774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.817738    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:16:36.825445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.825414    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-775f8pd26"]
Nov 02 08:16:36.869845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.869812    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:16:36.876232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:36.876203    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-cc9fsvdrq"]
Nov 02 08:16:37.020148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.020069    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:16:37.020859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.020844    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8788fb76-0547-49a9-90bc-c945cef42e9a" containerName="payload-processing"
Nov 02 08:16:37.020906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.020860    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8788fb76-0547-49a9-90bc-c945cef42e9a" containerName="payload-processing"
Nov 02 08:16:37.020906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.020874    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" containerName="payload-processing"
Nov 02 08:16:37.020906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.020880    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" containerName="payload-processing"
Nov 02 08:16:37.021018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.021005    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8788fb76-0547-49a9-90bc-c945cef42e9a" containerName="payload-processing"
Nov 02 08:16:37.021018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.021013    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" containerName="payload-processing"
Nov 02 08:16:37.026431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.026411    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.043174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.043150    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-78ddbc\""
Nov 02 08:16:37.043393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.043371    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-78ddbc-dockercfg-pls6g\""
Nov 02 08:16:37.045078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.045046    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:16:37.057721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.057688    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sht7b\" (UniqueName: \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.057871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.057766    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.211463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.200297    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sht7b\" (UniqueName: \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.211463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.204455    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.211463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.205550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.234619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.234586    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:16:37.241790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.241764    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.298280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.297429    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sht7b\" (UniqueName: \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\") pod \"payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.380522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.380487    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:37.400488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.399978    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4839b5-0ab8-4fb4-b923-c81860b061d7" path="/var/lib/kubelet/pods/7e4839b5-0ab8-4fb4-b923-c81860b061d7/volumes"
Nov 02 08:16:37.400751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.400555    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8788fb76-0547-49a9-90bc-c945cef42e9a" path="/var/lib/kubelet/pods/8788fb76-0547-49a9-90bc-c945cef42e9a/volumes"
Nov 02 08:16:37.400935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.400916    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:16:37.480850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.480807    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.481160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.481140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjxz\" (UniqueName: \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.582000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.581915    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.582159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.582116    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjxz\" (UniqueName: \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.582845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.582815    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.606524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.606487    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjxz\" (UniqueName: \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\") pod \"payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.694412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.694382    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:37.703054 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:37.703026    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672d5549_1205_4fb8_a1e6_56a2971e41c9.slice/crio-90d618b58abfe062376f0a155c7bdef09d4243ef92adb48ebc7161cb655f6cfd WatchSource:0}: Error finding container 90d618b58abfe062376f0a155c7bdef09d4243ef92adb48ebc7161cb655f6cfd: Status 404 returned error can't find the container with id 90d618b58abfe062376f0a155c7bdef09d4243ef92adb48ebc7161cb655f6cfd
Nov 02 08:16:37.703904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.703881    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:16:37.776144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:37.776100    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" event={"ID":"672d5549-1205-4fb8-a1e6-56a2971e41c9","Type":"ContainerStarted","Data":"90d618b58abfe062376f0a155c7bdef09d4243ef92adb48ebc7161cb655f6cfd"}
Nov 02 08:16:38.058272 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:38.058242    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a3a97f_9032_4471_8867_96590684a65e.slice/crio-cc3d65837cebcc863cd743e3fd7001d892fb7beb2fa848d387a625906487dc61 WatchSource:0}: Error finding container cc3d65837cebcc863cd743e3fd7001d892fb7beb2fa848d387a625906487dc61: Status 404 returned error can't find the container with id cc3d65837cebcc863cd743e3fd7001d892fb7beb2fa848d387a625906487dc61
Nov 02 08:16:38.059914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.059888    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:16:38.785209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.785170    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" event={"ID":"48a3a97f-9032-4471-8867-96590684a65e","Type":"ContainerStarted","Data":"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"}
Nov 02 08:16:38.785506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.785240    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" event={"ID":"48a3a97f-9032-4471-8867-96590684a65e","Type":"ContainerStarted","Data":"cc3d65837cebcc863cd743e3fd7001d892fb7beb2fa848d387a625906487dc61"}
Nov 02 08:16:38.785506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.785273    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:38.786897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.786865    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" event={"ID":"672d5549-1205-4fb8-a1e6-56a2971e41c9","Type":"ContainerStarted","Data":"687ea352bac1ba44e5483ef39f50d99f6e72d4f7ba9c5327e5bb8d5518fd0514"}
Nov 02 08:16:38.787083 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.787065    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:38.827235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.827187    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" podStartSLOduration=1.827173956 podStartE2EDuration="1.827173956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:38.825951035 +0000 UTC m=+3143.588387943" watchObservedRunningTime="2026-11-02 08:16:38.827173956 +0000 UTC m=+3143.589610862"
Nov 02 08:16:38.867526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:38.867479    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" podStartSLOduration=2.867464316 podStartE2EDuration="2.867464316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:38.866993943 +0000 UTC m=+3143.629430849" watchObservedRunningTime="2026-11-02 08:16:38.867464316 +0000 UTC m=+3143.629901222"
Nov 02 08:16:47.725033 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:16:47.724995    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:16:49.801118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:49.801085    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:16:49.801621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:49.801221    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:16:54.380670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.380632    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:16:54.385388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.385370    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.410717 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.410677    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:16:54.413783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.413753    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b72fa3-dockercfg-tmnbw\""
Nov 02 08:16:54.416378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.414101    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b72fa3\""
Nov 02 08:16:54.549331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.548864    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dbz\" (UniqueName: \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.549331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.548973    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.611334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.610633    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:16:54.685283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.675826    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dbz\" (UniqueName: \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.685601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.684609    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:16:54.685727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.684702    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:54.686605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.686515    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.690375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.687400    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.709415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.706271    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dbz\" (UniqueName: \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\") pod \"payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:54.891283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.891249    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:54.891532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.891509    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqhc\" (UniqueName: \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:54.993495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.993416    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqhc\" (UniqueName: \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:54.993653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.993610    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:54.994394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:54.994373    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:55.001160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.001138    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:55.017092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.016976    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqhc\" (UniqueName: \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\") pod \"payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:55.308945 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:55.308904    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e1e52e_7379_4863_b1e3_3dfa3d1fc39a.slice/crio-5619763e394548dcf8268b63cdf80fd71aff82786a33f5bf0ec5c20384224ffd WatchSource:0}: Error finding container 5619763e394548dcf8268b63cdf80fd71aff82786a33f5bf0ec5c20384224ffd: Status 404 returned error can't find the container with id 5619763e394548dcf8268b63cdf80fd71aff82786a33f5bf0ec5c20384224ffd
Nov 02 08:16:55.309858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.309829    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:16:55.314028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.314001    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:55.632572 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:16:55.632533    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1292b7_fafb_42ce_b43c_fe82b356ca0b.slice/crio-1372fc848f72b4a29217ceb43123ad578308604aed2d35d1d52655cb8d69d161 WatchSource:0}: Error finding container 1372fc848f72b4a29217ceb43123ad578308604aed2d35d1d52655cb8d69d161: Status 404 returned error can't find the container with id 1372fc848f72b4a29217ceb43123ad578308604aed2d35d1d52655cb8d69d161
Nov 02 08:16:55.634597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.634572    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:16:55.989384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.989274    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" event={"ID":"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a","Type":"ContainerStarted","Data":"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"}
Nov 02 08:16:55.989384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.989366    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" event={"ID":"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a","Type":"ContainerStarted","Data":"5619763e394548dcf8268b63cdf80fd71aff82786a33f5bf0ec5c20384224ffd"}
Nov 02 08:16:55.989652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.989398    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:16:55.991268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.991241    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" event={"ID":"ac1292b7-fafb-42ce-b43c-fe82b356ca0b","Type":"ContainerStarted","Data":"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"}
Nov 02 08:16:55.991423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.991284    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" event={"ID":"ac1292b7-fafb-42ce-b43c-fe82b356ca0b","Type":"ContainerStarted","Data":"1372fc848f72b4a29217ceb43123ad578308604aed2d35d1d52655cb8d69d161"}
Nov 02 08:16:55.991423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:55.991339    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:16:56.032531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:56.032476    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" podStartSLOduration=2.032462397 podStartE2EDuration="2.032462397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:56.030271718 +0000 UTC m=+3160.792708623" watchObservedRunningTime="2026-11-02 08:16:56.032462397 +0000 UTC m=+3160.794899302"
Nov 02 08:16:56.072488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:16:56.072439    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" podStartSLOduration=2.072425095 podStartE2EDuration="2.072425095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:56.070963664 +0000 UTC m=+3160.833400571" watchObservedRunningTime="2026-11-02 08:16:56.072425095 +0000 UTC m=+3160.834862014"
Nov 02 08:17:07.002408 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:07.002377    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:17:07.002876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:07.002447    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:17:11.987432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:11.987393    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:17:17.725033 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:17:17.724995    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:17:28.560061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.559976    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:17:28.566054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.566033    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.586376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.586344    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:17:28.592838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.592808    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-78ddbc-serving-cert\""
Nov 02 08:17:28.670968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.670924    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.671144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.671065    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.671326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.671293    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndwp\" (UniqueName: \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.671445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.671426    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.773176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.773137    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndwp\" (UniqueName: \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.773418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.773230    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.773418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.773267    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.773418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.773349    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.774076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.774053    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.775876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.775851    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.775975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.775878    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.796911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.796877    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndwp\" (UniqueName: \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\") pod \"maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:28.888381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:28.886580    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:29.226827 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:17:29.226781    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c49db03_b0bc_41e8_92fc_8d9fdec4d6b3.slice/crio-5348cf3b9bdc5f673aa2018992487b568006c6fc7dadf559db798d3435387d50 WatchSource:0}: Error finding container 5348cf3b9bdc5f673aa2018992487b568006c6fc7dadf559db798d3435387d50: Status 404 returned error can't find the container with id 5348cf3b9bdc5f673aa2018992487b568006c6fc7dadf559db798d3435387d50
Nov 02 08:17:29.228397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:29.228370    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:17:29.377215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:29.377179    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" event={"ID":"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3","Type":"ContainerStarted","Data":"5348cf3b9bdc5f673aa2018992487b568006c6fc7dadf559db798d3435387d50"}
Nov 02 08:17:30.381142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:30.381087    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" event={"ID":"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3","Type":"ContainerStarted","Data":"1ee74ab5a89043c82524ed9fc6fbacdd793604f1f8c79fe1e62ec9a41e5fb3cf"}
Nov 02 08:17:30.381560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:30.381160    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:30.423804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:30.423750    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" podStartSLOduration=2.037685882 podStartE2EDuration="2.423736534s" totalImagesPullingTime="386.050652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:30.423420425 +0000 UTC m=+3195.185857331" watchObservedRunningTime="2026-11-02 08:17:30.423736534 +0000 UTC m=+3195.186173441"
Nov 02 08:17:36.394082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:36.394048    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:17:46.954283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:46.954248    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:17:46.960354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:46.960303    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.000501 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.000466    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b72fa3-serving-cert\""
Nov 02 08:17:47.008026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.007991    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:17:47.029811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.029775    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.030017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.029839    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.030017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.029943    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r978l\" (UniqueName: \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.030017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.029973    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.131916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.131879    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.132136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.131952    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.132136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.132030    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r978l\" (UniqueName: \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.132136 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:17:47.132038    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-serving-cert: secret "maas-api-e2e-inf-a-b72fa3-serving-cert" not found
Nov 02 08:17:47.132136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.132061    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.132136 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:17:47.132100    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls podName:ae26a955-a6d1-4db2-8772-fdb53c2d6d20 nodeName:}" failed. No retries permitted until 2026-11-02 08:17:47.632083025 +0000 UTC m=+3212.394519909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls") pod "maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" (UID: "ae26a955-a6d1-4db2-8772-fdb53c2d6d20") : secret "maas-api-e2e-inf-a-b72fa3-serving-cert" not found
Nov 02 08:17:47.132753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.132729    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.134787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.134757    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.197234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.197184    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r978l\" (UniqueName: \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.640034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.639994    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.642507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.642479    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") pod \"maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:47.724383 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:17:47.724343    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:17:47.876255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:47.876221    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:48.189854 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:17:48.189827    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae26a955_a6d1_4db2_8772_fdb53c2d6d20.slice/crio-c4501f0273abf6c471019e3448a840e996872858f3230c79115ffc6e80b42667 WatchSource:0}: Error finding container c4501f0273abf6c471019e3448a840e996872858f3230c79115ffc6e80b42667: Status 404 returned error can't find the container with id c4501f0273abf6c471019e3448a840e996872858f3230c79115ffc6e80b42667
Nov 02 08:17:48.193059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:48.193030    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:17:48.516297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:48.516259    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" event={"ID":"ae26a955-a6d1-4db2-8772-fdb53c2d6d20","Type":"ContainerStarted","Data":"c4501f0273abf6c471019e3448a840e996872858f3230c79115ffc6e80b42667"}
Nov 02 08:17:49.526720 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:49.526645    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" event={"ID":"ae26a955-a6d1-4db2-8772-fdb53c2d6d20","Type":"ContainerStarted","Data":"a3c6b3aaba524637365e3b60e0201e475184027280452564c6046fd6f01ae168"}
Nov 02 08:17:49.527186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:49.526796    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:49.573555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:49.573506    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" podStartSLOduration=3.178109158 podStartE2EDuration="3.573490743s" totalImagesPullingTime="395.381585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:49.568972212 +0000 UTC m=+3214.331409117" watchObservedRunningTime="2026-11-02 08:17:49.573490743 +0000 UTC m=+3214.335927649"
Nov 02 08:17:55.539733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:55.539701    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:17:59.027616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:17:59.025125    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:18:10.010887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.010851    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:18:10.011434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.011066    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy" containerID="cri-o://4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba" gracePeriod=30
Nov 02 08:18:10.857458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.857365    4515 generic.go:356] "Generic (PLEG): container finished" podID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerID="85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47" exitCode=0
Nov 02 08:18:10.857458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.857436    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerDied","Data":"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"}
Nov 02 08:18:10.857932 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.857915    4515 scope.go:122] "RemoveContainer" containerID="85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"
Nov 02 08:18:10.859976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.859948    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:10.860053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.860021    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:10.860095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:10.860053    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:11.868740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:11.868704    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerStarted","Data":"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"}
Nov 02 08:18:12.370056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:12.370026    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:18:12.370247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:12.370181    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 08:18:12.370247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:12.370232    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 08:18:13.378469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:13.378428    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 08:18:13.379096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:13.379059    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 08:18:14.369573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:14.369526    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 08:18:14.369795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:14.369592    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 08:18:15.371729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.371691    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 08:18:15.372198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.371752    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 08:18:15.404960 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.404934    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:18:15.452259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452222    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452359    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452417    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452628 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452539    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452636    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452683    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452712    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:18:15.452852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452744    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:18:15.452852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452839    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.452962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452871    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:18:15.452962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452911    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.453079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452985    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\") pod \"03e276e2-c6f8-4fad-9c35-f898381be2de\" (UID: \"03e276e2-c6f8-4fad-9c35-f898381be2de\") "
Nov 02 08:18:15.453079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.453044    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:18:15.453212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.452924    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:18:15.454253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.454154    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.454253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.454180    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.454253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.454194    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.454253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.454209    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e276e2-c6f8-4fad-9c35-f898381be2de-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.454253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.454223    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.455794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.455628    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:18:15.455794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.455765    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:18:15.456675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.456646    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:18:15.459745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.459723    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v" pod "03e276e2-c6f8-4fad-9c35-f898381be2de" (UID: "03e276e2-c6f8-4fad-9c35-f898381be2de"). InnerVolumeSpecName "kube-api-access-9tr9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:18:15.555811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.555776    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tr9v\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-kube-api-access-9tr9v\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.555811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.555807    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.555811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.555818    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.556049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.555826    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e276e2-c6f8-4fad-9c35-f898381be2de-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:18:15.900560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.900523    4515 generic.go:356] "Generic (PLEG): container finished" podID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerID="4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba" exitCode=0
Nov 02 08:18:15.900741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.900577    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"
Nov 02 08:18:15.900741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.900583    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" event={"ID":"03e276e2-c6f8-4fad-9c35-f898381be2de","Type":"ContainerDied","Data":"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"}
Nov 02 08:18:15.900741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.900636    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l" event={"ID":"03e276e2-c6f8-4fad-9c35-f898381be2de","Type":"ContainerDied","Data":"ba5ddc5c9217641e859649ea3d0b296e7fb8937a6ca22800e9ce30f7e3259e3c"}
Nov 02 08:18:15.900741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.900667    4515 scope.go:122] "RemoveContainer" containerID="4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"
Nov 02 08:18:15.913277 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.913253    4515 scope.go:122] "RemoveContainer" containerID="4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"
Nov 02 08:18:15.913588 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:18:15.913566    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba\": container with ID starting with 4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba not found: ID does not exist" containerID="4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"
Nov 02 08:18:15.913640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.913598    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba"} err="failed to get container status \"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba\": rpc error: code = NotFound desc = could not find container \"4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba\": container with ID starting with 4dd0c5ae94b8f87130ac4c80f48a374830424c4b1abfb29303669837813bc9ba not found: ID does not exist"
Nov 02 08:18:15.972015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.971978    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:18:15.984840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:15.984802    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ckx7l"]
Nov 02 08:18:16.370192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:16.370156    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 08:18:16.370417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:16.370214    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 08:18:17.377192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:17.377154    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" path="/var/lib/kubelet/pods/03e276e2-c6f8-4fad-9c35-f898381be2de/volumes"
Nov 02 08:18:17.377942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:17.377922    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:18:17.378361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:17.378337    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:18:17.379163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:17.379143    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:18:17.737541 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:18:17.737447    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:18:33.589210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.589169    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:18:33.591093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.591055    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy"
Nov 02 08:18:33.591093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.591090    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy"
Nov 02 08:18:33.591389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.591372    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e276e2-c6f8-4fad-9c35-f898381be2de" containerName="istio-proxy"
Nov 02 08:18:33.604202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.603126    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.623192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.622557    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d042a8-openshift-default-dockercfg-dgvtx\""
Nov 02 08:18:33.627732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.627693    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:18:33.679070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679021    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679164    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679221    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679353    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679435    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679780 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679706    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbt2\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679779    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.679848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.679837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.792116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.792070    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.792445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.792421    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbt2\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.792591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.792576    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.792730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.792714    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.792901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.792884    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.793039 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.793025    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.793174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.793161    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.793299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.793285    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.793443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.793425    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.794523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.794497    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.807544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.794939    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.807748 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.804906    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.807864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.805146    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.808009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.805457    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.835898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.835819    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.844007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.843927    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.857334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.846036    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbt2\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.873348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.871919    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:33.926908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:33.926864    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:34.254727 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:18:34.254687    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024ae50a_2682_4d40_9d8f_fbf460cdf97a.slice/crio-20db5d238876fc6aacf8b9c7cc699b0803a75a6e01bf5d671da9d51b4941ce5c WatchSource:0}: Error finding container 20db5d238876fc6aacf8b9c7cc699b0803a75a6e01bf5d671da9d51b4941ce5c: Status 404 returned error can't find the container with id 20db5d238876fc6aacf8b9c7cc699b0803a75a6e01bf5d671da9d51b4941ce5c
Nov 02 08:18:34.256486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:34.256346    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:18:34.257912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:34.257875    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:34.258026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:34.257974    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:34.258026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:34.258009    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:35.110002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:35.109960    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerStarted","Data":"09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b"}
Nov 02 08:18:35.110543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:35.110030    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerStarted","Data":"20db5d238876fc6aacf8b9c7cc699b0803a75a6e01bf5d671da9d51b4941ce5c"}
Nov 02 08:18:35.171287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:35.171229    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podStartSLOduration=2.171214309 podStartE2EDuration="2.171214309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:35.168735795 +0000 UTC m=+3259.931172700" watchObservedRunningTime="2026-11-02 08:18:35.171214309 +0000 UTC m=+3259.933651214"
Nov 02 08:18:35.927021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:35.926985    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:35.933202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:35.933177    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:36.117920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:36.117888    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:36.118698 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:36.118677    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:18:39.772337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.770467    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:18:39.789925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.789888    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.792107 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.792070    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:18:39.820989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.820953    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d042a8-dockercfg-rhr2h\""
Nov 02 08:18:39.821181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.821032    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d042a8\""
Nov 02 08:18:39.871130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.871091    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.871361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.871169    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rx74\" (UniqueName: \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.973173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.973117    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rx74\" (UniqueName: \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.973443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.973430    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.974217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.974199    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:39.997204 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:39.997168    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rx74\" (UniqueName: \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\") pod \"payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:40.136781 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:40.136698    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:40.442408 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:18:40.442370    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d05e55_eb9e_4032_89bd_32e65237f7eb.slice/crio-13772a48a9ff1be816c0083979ebff8c3eda22ae50f2b60b8548ff345e9c3cff WatchSource:0}: Error finding container 13772a48a9ff1be816c0083979ebff8c3eda22ae50f2b60b8548ff345e9c3cff: Status 404 returned error can't find the container with id 13772a48a9ff1be816c0083979ebff8c3eda22ae50f2b60b8548ff345e9c3cff
Nov 02 08:18:40.444205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:40.444177    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:18:41.186840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:41.186791    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" event={"ID":"b9d05e55-eb9e-4032-89bd-32e65237f7eb","Type":"ContainerStarted","Data":"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"}
Nov 02 08:18:41.187875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:41.186917    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" event={"ID":"b9d05e55-eb9e-4032-89bd-32e65237f7eb","Type":"ContainerStarted","Data":"13772a48a9ff1be816c0083979ebff8c3eda22ae50f2b60b8548ff345e9c3cff"}
Nov 02 08:18:41.187875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:41.187459    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:41.232112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:41.232059    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" podStartSLOduration=2.232046132 podStartE2EDuration="2.232046132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:41.228885549 +0000 UTC m=+3265.991322456" watchObservedRunningTime="2026-11-02 08:18:41.232046132 +0000 UTC m=+3265.994483071"
Nov 02 08:18:47.724925 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:18:47.724882    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:18:52.198094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:52.198058    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:18:56.922956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.922919    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:18:56.928078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.928052    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:56.946133 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.946101    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-eb6010\""
Nov 02 08:18:56.948253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.948228    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:18:56.955390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.955359    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-eb6010-dockercfg-659hq\""
Nov 02 08:18:56.992282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.992243    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:56.992699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.992679    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5nf\" (UniqueName: \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:56.996385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:56.996362    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:18:57.002571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.002554    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.040835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.025521    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:18:57.191385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.152844    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.191804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.191776    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4b4\" (UniqueName: \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.192130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.192111    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5nf\" (UniqueName: \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:57.192540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.192507    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:57.223576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.223544    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:57.254218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.254162    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5nf\" (UniqueName: \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\") pod \"payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:57.297459 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.294806    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.297673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.296099    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.297673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.297609    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4b4\" (UniqueName: \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.341990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.341955    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4b4\" (UniqueName: \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\") pod \"payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") " pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.380603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.380574    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:57.543908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.543873    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:57.700410 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:18:57.700375    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6a1263_e25f_430c_a3eb_9817ff0ca3ec.slice/crio-fb22336fc08df52606ad05b031e1ce7029e65c36e456d3860d353a33a976d724 WatchSource:0}: Error finding container fb22336fc08df52606ad05b031e1ce7029e65c36e456d3860d353a33a976d724: Status 404 returned error can't find the container with id fb22336fc08df52606ad05b031e1ce7029e65c36e456d3860d353a33a976d724
Nov 02 08:18:57.700954 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.700925    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:18:57.851778 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:18:57.851745    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3408354_361c_4d06_ad38_b3271fbc7d25.slice/crio-e969b9aa173fbb812feb530c267f99d34928e7e9a1789e1634a7dd3c414b4122 WatchSource:0}: Error finding container e969b9aa173fbb812feb530c267f99d34928e7e9a1789e1634a7dd3c414b4122: Status 404 returned error can't find the container with id e969b9aa173fbb812feb530c267f99d34928e7e9a1789e1634a7dd3c414b4122
Nov 02 08:18:57.855374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:57.855348    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:18:58.354175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.354137    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" event={"ID":"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec","Type":"ContainerStarted","Data":"2ed6438774a9b5024cc38f3610de31a0019c8f704f032cefab9205c55f5dc88f"}
Nov 02 08:18:58.354674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.354211    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:18:58.354674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.354238    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" event={"ID":"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec","Type":"ContainerStarted","Data":"fb22336fc08df52606ad05b031e1ce7029e65c36e456d3860d353a33a976d724"}
Nov 02 08:18:58.355938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.355910    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" event={"ID":"f3408354-361c-4d06-ad38-b3271fbc7d25","Type":"ContainerStarted","Data":"d129c2dd6bc7400bcc1f403f17442e1d388dc815afb2fd264134f8508e15f400"}
Nov 02 08:18:58.356085 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.355961    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" event={"ID":"f3408354-361c-4d06-ad38-b3271fbc7d25","Type":"ContainerStarted","Data":"e969b9aa173fbb812feb530c267f99d34928e7e9a1789e1634a7dd3c414b4122"}
Nov 02 08:18:58.356085 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.356009    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:18:58.464881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.464827    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" podStartSLOduration=2.46481336 podStartE2EDuration="2.46481336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:58.460846359 +0000 UTC m=+3283.223283265" watchObservedRunningTime="2026-11-02 08:18:58.46481336 +0000 UTC m=+3283.227250265"
Nov 02 08:18:58.523281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:18:58.523228    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" podStartSLOduration=2.523210187 podStartE2EDuration="2.523210187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:58.520043729 +0000 UTC m=+3283.282480637" watchObservedRunningTime="2026-11-02 08:18:58.523210187 +0000 UTC m=+3283.285647097"
Nov 02 08:19:09.380663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:19:09.380630    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:19:09.381082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:19:09.380884    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:19:09.736761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:19:09.736652    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:19:17.724713 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:19:17.724676    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:19:47.725391 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:19:47.725349    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:20:03.966248 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:03.966210    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:20:03.974357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:03.974325    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:03.993426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:03.993390    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:20:04.003931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.003890    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6f4fe\"/\"test-model-e6f4fe-kserve-self-signed-certs\""
Nov 02 08:20:04.005761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.005733    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6f4fe\"/\"default-dockercfg-26fx5\""
Nov 02 08:20:04.006098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.006077    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6f4fe\"/\"openshift-service-ca.crt\""
Nov 02 08:20:04.006396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.006379    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6f4fe\"/\"kube-root-ca.crt\""
Nov 02 08:20:04.028365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.028328    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.030074 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.030041    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.030235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.030211    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.030302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.030252    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.030418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.030360    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.030516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.030491    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.131975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.131937    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.131975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.131981    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132078    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132115    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132219    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132245    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132523    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132634 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132579    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.132678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.132627    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.134413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.134387    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.134692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.134677    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.156282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.156249    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\") pod \"test-model-e6f4fe-kserve-df77f5479-72cr4\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") " pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.298429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.297946    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:04.653783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.653741    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:20:04.654414 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:20:04.654383    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5c4923_2d75_4e86_92bb_9f93a408acaf.slice/crio-40160ce1cac24de2f7a7efae2b9daae1e00cf041fa87868709b09192f4774bec WatchSource:0}: Error finding container 40160ce1cac24de2f7a7efae2b9daae1e00cf041fa87868709b09192f4774bec: Status 404 returned error can't find the container with id 40160ce1cac24de2f7a7efae2b9daae1e00cf041fa87868709b09192f4774bec
Nov 02 08:20:04.970772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.970681    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" event={"ID":"1c5c4923-2d75-4e86-92bb-9f93a408acaf","Type":"ContainerStarted","Data":"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"}
Nov 02 08:20:04.970772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:04.970731    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" event={"ID":"1c5c4923-2d75-4e86-92bb-9f93a408acaf","Type":"ContainerStarted","Data":"40160ce1cac24de2f7a7efae2b9daae1e00cf041fa87868709b09192f4774bec"}
Nov 02 08:20:05.015218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:05.015158    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" podStartSLOduration=2.015139061 podStartE2EDuration="2.015139061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:05.013156566 +0000 UTC m=+3349.775593471" watchObservedRunningTime="2026-11-02 08:20:05.015139061 +0000 UTC m=+3349.777575967"
Nov 02 08:20:05.299041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:05.299000    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:14.299425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:14.299386    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:14.312130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:14.312094    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:15.071528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:15.071497    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:20:17.724656 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:20:17.724617    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:20:23.136335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:23.127857    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:20:35.421337 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:20:35.405220    4515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024ae50a_2682_4d40_9d8f_fbf460cdf97a.slice/crio-09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024ae50a_2682_4d40_9d8f_fbf460cdf97a.slice/crio-conmon-09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 08:20:36.119180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.119142    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:36.119386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.119224    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:36.313249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.313210    4515 generic.go:356] "Generic (PLEG): container finished" podID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerID="09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b" exitCode=0
Nov 02 08:20:36.313468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.313285    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerDied","Data":"09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b"}
Nov 02 08:20:36.313836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.313821    4515 scope.go:122] "RemoveContainer" containerID="09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b"
Nov 02 08:20:36.315980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.315948    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:36.316101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.316022    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:36.316101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:36.316058    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:37.323568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:37.323524    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerStarted","Data":"0fb9881d238265e706acab165d408b7a138637fdc66840e3e3f8665c121cb671"}
Nov 02 08:20:37.933276 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:37.933243    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:20:37.933940 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:37.933913    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:37.941381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:37.934126    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:38.935784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:38.935745    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:38.936428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:38.936393    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:39.938333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:39.935737    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:39.938333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:39.935807    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:40.933333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:40.929695    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:40.933333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:40.929763    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:41.937334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:41.932734    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:41.937334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:41.932809    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:42.928331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:42.927714    4515 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 02 08:20:42.928331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:42.927809    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 02 08:20:43.933233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:43.933195    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:20:43.933726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:43.933707    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:20:43.934571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:43.934546    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:20:47.724884 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:20:47.724840    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:20:50.334887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.334836    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:20:50.341359 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.341290    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.358019 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.357987    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:20:50.364171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.364146    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d042a8\"/\"default-dockercfg-bl4jb\""
Nov 02 08:20:50.364320 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.364257    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d042a8\"/\"test-model-d042a8-kserve-self-signed-certs\""
Nov 02 08:20:50.368998 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.368975    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d042a8\"/\"kube-root-ca.crt\""
Nov 02 08:20:50.369143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.369068    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d042a8\"/\"openshift-service-ca.crt\""
Nov 02 08:20:50.469801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.469761    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.470013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.469853    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.470013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.469921    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.470159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.470078    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.470265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.470246    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knddg\" (UniqueName: \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.470791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.470587    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.572633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572597    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.572850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572657    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.572850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572681    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.572850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572725    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.572850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572790    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.573068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.572923    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knddg\" (UniqueName: \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.573068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.573043    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.573188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.573104    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.573365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.573345    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.575214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.575185    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.575383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.575329    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.596962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.596900    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knddg\" (UniqueName: \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\") pod \"test-model-d042a8-kserve-b68d88c7-7mdrb\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") " pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:50.668166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:50.667877    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:20:51.052207 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:20:51.052178    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac7b209_3d01_49a5_9961_4a2dbcc799b8.slice/crio-34135273abea659126a7713b6325a86b7d04a405aeca6b52362b15f58dd4b916 WatchSource:0}: Error finding container 34135273abea659126a7713b6325a86b7d04a405aeca6b52362b15f58dd4b916: Status 404 returned error can't find the container with id 34135273abea659126a7713b6325a86b7d04a405aeca6b52362b15f58dd4b916
Nov 02 08:20:51.054348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:51.054267    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:20:51.536371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:51.536335    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" event={"ID":"7ac7b209-3d01-49a5-9961-4a2dbcc799b8","Type":"ContainerStarted","Data":"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"}
Nov 02 08:20:51.536371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:51.536377    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" event={"ID":"7ac7b209-3d01-49a5-9961-4a2dbcc799b8","Type":"ContainerStarted","Data":"34135273abea659126a7713b6325a86b7d04a405aeca6b52362b15f58dd4b916"}
Nov 02 08:20:51.584051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:51.583991    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" podStartSLOduration=1.5839717869999999 podStartE2EDuration="1.583971787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:51.578556683 +0000 UTC m=+3396.340993593" watchObservedRunningTime="2026-11-02 08:20:51.583971787 +0000 UTC m=+3396.346408700"
Nov 02 08:20:51.669381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:20:51.669342    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:21:00.669546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:00.669501    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:21:00.682901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:00.682872    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:21:01.659475 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:01.659440    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:21:11.568903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:11.568865    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:21:17.742880 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:21:17.742835    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:21:36.629733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.629702    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:21:36.634826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.634808    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.663222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.663190    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ba44ed-openshift-default-dockercfg-f6f29\""
Nov 02 08:21:36.667661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.667636    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:21:36.788843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.788803    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.788867    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.788948    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjjp7\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789094    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789191    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789324    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789394    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789472    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.789537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.789528    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.891662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891571    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.891662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891644    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.891908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891681    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjjp7\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.891908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891763    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.891908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891867    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.891968    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892009    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892099    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892111    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892185    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892208    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892536 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892514    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892533    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.892700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.892671    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.894632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.894613    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.894882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.894854    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.915672 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.915649    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.915850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.915833    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjjp7\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:36.950164 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:36.950134    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:37.294018 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:21:37.293970    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702facf3_39ec_4bb2_ba2c_c898a6f9dc15.slice/crio-b6f8e8f54babe42c2874f770a3ebbc7a3de6a76a3f5a5ec5cd384461464052ec WatchSource:0}: Error finding container b6f8e8f54babe42c2874f770a3ebbc7a3de6a76a3f5a5ec5cd384461464052ec: Status 404 returned error can't find the container with id b6f8e8f54babe42c2874f770a3ebbc7a3de6a76a3f5a5ec5cd384461464052ec
Nov 02 08:21:37.295934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.295901    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:21:37.299336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.299278    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:37.299467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.299389    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:37.299467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.299437    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:37.990383 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.990340    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" event={"ID":"702facf3-39ec-4bb2-ba2c-c898a6f9dc15","Type":"ContainerStarted","Data":"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"}
Nov 02 08:21:37.990858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:37.990407    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" event={"ID":"702facf3-39ec-4bb2-ba2c-c898a6f9dc15","Type":"ContainerStarted","Data":"b6f8e8f54babe42c2874f770a3ebbc7a3de6a76a3f5a5ec5cd384461464052ec"}
Nov 02 08:21:38.049327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:38.049258    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podStartSLOduration=2.049241749 podStartE2EDuration="2.049241749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:38.046345235 +0000 UTC m=+3442.808782142" watchObservedRunningTime="2026-11-02 08:21:38.049241749 +0000 UTC m=+3442.811678713"
Nov 02 08:21:38.950888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:38.950855    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:38.952781 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:38.952756    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:38.952911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:38.952808    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:39.950828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:39.950794    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:39.951217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:39.950865    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:40.955336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:40.954007    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:40.955336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:40.954113    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:41.951569 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:41.951508    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:41.951809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:41.951581    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:42.950779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:42.950746    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:42.951258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:42.950804    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:43.951072 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:43.951032    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 02 08:21:43.951619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:43.951120    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 02 08:21:44.955841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:44.955808    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:44.956287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:44.956275    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:44.956987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:44.956964    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:21:47.726636 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:21:47.726597    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:21:56.168465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:56.168427    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 08:21:56.168968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:56.168778    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" containerID="cri-o://00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f" gracePeriod=30
Nov 02 08:21:58.837929 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:58.837845    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 08:21:58.838364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:21:58.838051    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" containerID="cri-o://deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b" gracePeriod=30
Nov 02 08:22:01.089841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.089799    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:22:01.090350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.089877    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:22:01.545746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.545719    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 08:22:01.616692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.616600    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.616692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.616680    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.616937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.616857    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.616984    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:01.617037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617018    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617171    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617205    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617220    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:01.617416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617269    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:01.617416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617278    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.617946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617370    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\") pod \"151f5506-8b72-459f-ba40-4479c93ca1c5\" (UID: \"151f5506-8b72-459f-ba40-4479c93ca1c5\") "
Nov 02 08:22:01.618188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617794    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:01.618188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.617808    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:01.618811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.618780    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.618811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.618807    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.618963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.618820    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.618963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.618834    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.618963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.618847    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f5506-8b72-459f-ba40-4479c93ca1c5-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.620126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.620101    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:01.620257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.620182    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:01.620457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.620436    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:01.621349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.621125    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf" pod "151f5506-8b72-459f-ba40-4479c93ca1c5" (UID: "151f5506-8b72-459f-ba40-4479c93ca1c5"). InnerVolumeSpecName "kube-api-access-2xxwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:01.722557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.722403    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.722557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.722441    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.722557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.722456    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f5506-8b72-459f-ba40-4479c93ca1c5-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.722557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.722472    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxwf\" (UniqueName: \"kubernetes.io/projected/151f5506-8b72-459f-ba40-4479c93ca1c5-kube-api-access-2xxwf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:01.812487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.812446    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:22:01.812661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:01.812523    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:22:02.229094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.229060    4515 generic.go:356] "Generic (PLEG): container finished" podID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerID="00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f" exitCode=0
Nov 02 08:22:02.229590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.229120    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"
Nov 02 08:22:02.229590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.229146    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" event={"ID":"151f5506-8b72-459f-ba40-4479c93ca1c5","Type":"ContainerDied","Data":"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"}
Nov 02 08:22:02.229590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.229209    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq" event={"ID":"151f5506-8b72-459f-ba40-4479c93ca1c5","Type":"ContainerDied","Data":"b138f0f96a3f1632e81a23b3ec450a20bce01ce68635c55f933d79cad6a6956c"}
Nov 02 08:22:02.229590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.229231    4515 scope.go:122] "RemoveContainer" containerID="00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"
Nov 02 08:22:02.241551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.241528    4515 scope.go:122] "RemoveContainer" containerID="00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"
Nov 02 08:22:02.241853 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:02.241833    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f\": container with ID starting with 00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f not found: ID does not exist" containerID="00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"
Nov 02 08:22:02.241909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.241862    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f"} err="failed to get container status \"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f\": rpc error: code = NotFound desc = could not find container \"00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f\": container with ID starting with 00651e403be2506cc9a12aeee4ad9a077731cd884290f95aa66ba784683cea4f not found: ID does not exist"
Nov 02 08:22:02.287504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.287473    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 08:22:02.300254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:02.300222    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-8kbcq"]
Nov 02 08:22:03.377189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:03.377154    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" path="/var/lib/kubelet/pods/151f5506-8b72-459f-ba40-4479c93ca1c5/volumes"
Nov 02 08:22:04.129743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.129717    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 08:22:04.167772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.167738    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.167963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.167799    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.167963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.167856    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.167963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.167895    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.167963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.167947    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.168214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168020    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.168214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168202    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.168380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168213    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:04.168380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168223    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:04.168380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168281    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.168560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168405    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\") pod \"756d4e3e-a850-49eb-b8ff-0bde2e052020\" (UID: \"756d4e3e-a850-49eb-b8ff-0bde2e052020\") "
Nov 02 08:22:04.168560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168506    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:04.168560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168550    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:04.168729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.168698    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:04.169054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.169035    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d4e3e-a850-49eb-b8ff-0bde2e052020-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.169054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.169053    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.169210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.169062    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.169210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.169070    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.169210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.169080    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.170490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.170465    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:04.170589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.170512    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "kube-api-access-v2wq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:04.170589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.170571    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:04.170721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.170703    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy" pod "756d4e3e-a850-49eb-b8ff-0bde2e052020" (UID: "756d4e3e-a850-49eb-b8ff-0bde2e052020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:04.270449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.270418    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.270449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.270446    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.270449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.270455    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2wq8\" (UniqueName: \"kubernetes.io/projected/756d4e3e-a850-49eb-b8ff-0bde2e052020-kube-api-access-v2wq8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.270705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.270467    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d4e3e-a850-49eb-b8ff-0bde2e052020-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:04.275046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.275016    4515 generic.go:356] "Generic (PLEG): container finished" podID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerID="deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b" exitCode=0
Nov 02 08:22:04.275197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.275073    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"
Nov 02 08:22:04.275197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.275102    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" event={"ID":"756d4e3e-a850-49eb-b8ff-0bde2e052020","Type":"ContainerDied","Data":"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"}
Nov 02 08:22:04.275197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.275166    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl" event={"ID":"756d4e3e-a850-49eb-b8ff-0bde2e052020","Type":"ContainerDied","Data":"d934ef5a9dd2bce7c3d4752cf856b0280b613ec3a5e37d690d8f4a8a16b8878d"}
Nov 02 08:22:04.275197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.275190    4515 scope.go:122] "RemoveContainer" containerID="deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"
Nov 02 08:22:04.287623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.287595    4515 scope.go:122] "RemoveContainer" containerID="deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"
Nov 02 08:22:04.287986 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:04.287958    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b\": container with ID starting with deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b not found: ID does not exist" containerID="deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"
Nov 02 08:22:04.288086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.287993    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b"} err="failed to get container status \"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b\": rpc error: code = NotFound desc = could not find container \"deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b\": container with ID starting with deee1d513330b196d720adef1e2e6b5f2391a97fcf40233920411b41c8acfb8b not found: ID does not exist"
Nov 02 08:22:04.336804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.336759    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 08:22:04.347437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:04.347302    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5dpsl"]
Nov 02 08:22:05.375478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:05.375437    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" path="/var/lib/kubelet/pods/756d4e3e-a850-49eb-b8ff-0bde2e052020/volumes"
Nov 02 08:22:10.703065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:10.703031    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:22:10.703541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:10.703242    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy" containerID="cri-o://4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc" gracePeriod=30
Nov 02 08:22:10.716170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:10.716139    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:22:10.716378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:10.716356    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy" containerID="cri-o://882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38" gracePeriod=30
Nov 02 08:22:11.377767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:11.377735    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 08:22:11.377987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:11.377962    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" containerID="cri-o://6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d" gracePeriod=30
Nov 02 08:22:15.960410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:15.960286    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:22:15.960894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:15.960437    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:22:16.026659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.026349    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:22:16.054224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.054189    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:22:16.167093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167049    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.167093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167092    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.167393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167119    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.167393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167189    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.167393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167261    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167299    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167371    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167446    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167508    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167538    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:16.167692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167548    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.167994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.167968    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.168098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168077    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.168171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168126    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.168171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168154    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.168394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168372    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.168592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168428    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.168592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168476    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.168592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168522    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.168592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168560    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\") pod \"9b641edf-1f18-4f23-a30e-5d177cb1633e\" (UID: \"9b641edf-1f18-4f23-a30e-5d177cb1633e\") "
Nov 02 08:22:16.168851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168688    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\") pod \"189a9882-e40a-4271-bf60-01f888c4e043\" (UID: \"189a9882-e40a-4271-bf60-01f888c4e043\") "
Nov 02 08:22:16.168851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.168793    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:16.169058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169035    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.169647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169619    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b641edf-1f18-4f23-a30e-5d177cb1633e-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.169772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169651    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.169772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169665    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.169772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169678    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189a9882-e40a-4271-bf60-01f888c4e043-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.169772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.169693    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.170033 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.170011    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.170249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.170220    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.170551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.170529    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.170873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.170852    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.171008 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.170952    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.171845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.171790    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:16.172052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.172020    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:16.172939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.172750    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.172939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.172797    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "kube-api-access-hlhzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.173607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.173339    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "kube-api-access-vrdtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.173607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.173550    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy" pod "189a9882-e40a-4271-bf60-01f888c4e043" (UID: "189a9882-e40a-4271-bf60-01f888c4e043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.173860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.173832    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.174160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.174140    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token" pod "9b641edf-1f18-4f23-a30e-5d177cb1633e" (UID: "9b641edf-1f18-4f23-a30e-5d177cb1633e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277825    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277866    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277882    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277895    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277909    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhzj\" (UniqueName: \"kubernetes.io/projected/189a9882-e40a-4271-bf60-01f888c4e043-kube-api-access-hlhzj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277924    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277938    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189a9882-e40a-4271-bf60-01f888c4e043-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277952    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdtc\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-kube-api-access-vrdtc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277966    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277980    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.277993    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.278007    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b641edf-1f18-4f23-a30e-5d177cb1633e-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.278209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.278020    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189a9882-e40a-4271-bf60-01f888c4e043-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.373879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.373842    4515 generic.go:356] "Generic (PLEG): container finished" podID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerID="4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc" exitCode=0
Nov 02 08:22:16.374101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.373885    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"
Nov 02 08:22:16.374101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.373926    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" event={"ID":"9b641edf-1f18-4f23-a30e-5d177cb1633e","Type":"ContainerDied","Data":"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"}
Nov 02 08:22:16.374101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.373983    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr" event={"ID":"9b641edf-1f18-4f23-a30e-5d177cb1633e","Type":"ContainerDied","Data":"fdd495931a8d2904a92f22c3409d827d9dd6f59a0ba74e8d2dfa0b04aa296b0e"}
Nov 02 08:22:16.374101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.374007    4515 scope.go:122] "RemoveContainer" containerID="4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"
Nov 02 08:22:16.375891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.375861    4515 generic.go:356] "Generic (PLEG): container finished" podID="189a9882-e40a-4271-bf60-01f888c4e043" containerID="882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38" exitCode=0
Nov 02 08:22:16.376047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.375911    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"
Nov 02 08:22:16.376047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.375930    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" event={"ID":"189a9882-e40a-4271-bf60-01f888c4e043","Type":"ContainerDied","Data":"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"}
Nov 02 08:22:16.376047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.375974    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q" event={"ID":"189a9882-e40a-4271-bf60-01f888c4e043","Type":"ContainerDied","Data":"b32dd1b092fef056a4dc94712485e6180da544df88ad80ebb595abd73e01d9cc"}
Nov 02 08:22:16.396557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.396533    4515 scope.go:122] "RemoveContainer" containerID="4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"
Nov 02 08:22:16.396886 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:16.396867    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc\": container with ID starting with 4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc not found: ID does not exist" containerID="4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"
Nov 02 08:22:16.396945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.396895    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc"} err="failed to get container status \"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc\": rpc error: code = NotFound desc = could not find container \"4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc\": container with ID starting with 4d19ae3808ee81ffc717dda34346c8024835a55f3a10829790ae2bd3546588fc not found: ID does not exist"
Nov 02 08:22:16.396945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.396913    4515 scope.go:122] "RemoveContainer" containerID="882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"
Nov 02 08:22:16.408396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.408371    4515 scope.go:122] "RemoveContainer" containerID="882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"
Nov 02 08:22:16.408698 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:16.408679    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38\": container with ID starting with 882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38 not found: ID does not exist" containerID="882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"
Nov 02 08:22:16.408762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.408708    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38"} err="failed to get container status \"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38\": rpc error: code = NotFound desc = could not find container \"882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38\": container with ID starting with 882944df7b0716814d556f69114ddb85e0d104f02374573a4d089fcc644fba38 not found: ID does not exist"
Nov 02 08:22:16.432409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.432371    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:22:16.439162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.439134    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gsh6q"]
Nov 02 08:22:16.473096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.473062    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:22:16.476951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.476930    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8qdvr"]
Nov 02 08:22:16.648088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.648060    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 08:22:16.786749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.786714    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.786933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.786858    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.786933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.786924    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787011 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.786990    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787035    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787079    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787223 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787196    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787198    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.787417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787299    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787385    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\") pod \"c94421cd-cc31-4922-95fd-06d462d95d38\" (UID: \"c94421cd-cc31-4922-95fd-06d462d95d38\") "
Nov 02 08:22:16.787579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787554    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.787638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787600    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:16.787867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787845    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.787974 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.787863    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.788143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.788126    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.788218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.788145    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.788218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.788156    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94421cd-cc31-4922-95fd-06d462d95d38-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.788218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.788164    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.788218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.788173    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.789275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.789249    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:16.789390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.789363    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:16.789587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.789564    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.789675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.789599    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6" pod "c94421cd-cc31-4922-95fd-06d462d95d38" (UID: "c94421cd-cc31-4922-95fd-06d462d95d38"). InnerVolumeSpecName "kube-api-access-dlmj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:16.890342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.890248    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94421cd-cc31-4922-95fd-06d462d95d38-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.890342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.890288    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.890342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.890304    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94421cd-cc31-4922-95fd-06d462d95d38-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.890342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:16.890333    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmj6\" (UniqueName: \"kubernetes.io/projected/c94421cd-cc31-4922-95fd-06d462d95d38-kube-api-access-dlmj6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:17.394186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.394152    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189a9882-e40a-4271-bf60-01f888c4e043" path="/var/lib/kubelet/pods/189a9882-e40a-4271-bf60-01f888c4e043/volumes"
Nov 02 08:22:17.395352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.394990    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" path="/var/lib/kubelet/pods/9b641edf-1f18-4f23-a30e-5d177cb1633e/volumes"
Nov 02 08:22:17.402648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.402599    4515 generic.go:356] "Generic (PLEG): container finished" podID="c94421cd-cc31-4922-95fd-06d462d95d38" containerID="6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d" exitCode=0
Nov 02 08:22:17.402817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.402679    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" event={"ID":"c94421cd-cc31-4922-95fd-06d462d95d38","Type":"ContainerDied","Data":"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"}
Nov 02 08:22:17.402817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.402730    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv" event={"ID":"c94421cd-cc31-4922-95fd-06d462d95d38","Type":"ContainerDied","Data":"ea683a523dc805316d0a690c140125d384b197c499a6d9dfaf72e1c1cebad519"}
Nov 02 08:22:17.402817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.402762    4515 scope.go:122] "RemoveContainer" containerID="6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"
Nov 02 08:22:17.402997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.402920    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"
Nov 02 08:22:17.423469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.423434    4515 scope.go:122] "RemoveContainer" containerID="6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"
Nov 02 08:22:17.423785 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:17.423761    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d\": container with ID starting with 6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d not found: ID does not exist" containerID="6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"
Nov 02 08:22:17.423890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.423792    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d"} err="failed to get container status \"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d\": rpc error: code = NotFound desc = could not find container \"6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d\": container with ID starting with 6f972ed56f9ab71b0b550abf16a70d9580b4ccb4ada502e65050f706c211426d not found: ID does not exist"
Nov 02 08:22:17.472705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.472665    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 08:22:17.486344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:17.486281    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-qnwgv"]
Nov 02 08:22:17.732449 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:17.732356    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:22:19.376852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:19.376812    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" path="/var/lib/kubelet/pods/c94421cd-cc31-4922-95fd-06d462d95d38/volumes"
Nov 02 08:22:33.095105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:33.095064    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:22:47.726001 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:22:47.725956    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:22:49.667377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:49.667342    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:22:49.667875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:49.667614    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy" containerID="cri-o://c37389f33e77f55bffbfff0dc1c8d25092d2d53eca4612832dd3ae671280013b" gracePeriod=30
Nov 02 08:22:53.013830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:53.013797    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:22:54.900715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:54.900683    4515 generic.go:356] "Generic (PLEG): container finished" podID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerID="c37389f33e77f55bffbfff0dc1c8d25092d2d53eca4612832dd3ae671280013b" exitCode=0
Nov 02 08:22:54.901416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:54.901220    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerDied","Data":"c37389f33e77f55bffbfff0dc1c8d25092d2d53eca4612832dd3ae671280013b"}
Nov 02 08:22:54.901670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:54.901654    4515 scope.go:122] "RemoveContainer" containerID="bf423c24f53a8991894b5e7ff29f570f40893a8369729f326890a9e4680d28da"
Nov 02 08:22:55.020368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.020342    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098181    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098288    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098427    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098456    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098498    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098593    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098686    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098746    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.099801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.098785    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\") pod \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\" (UID: \"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b\") "
Nov 02 08:22:55.100929 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.100889    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:55.101056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.100990    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:55.101264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.101241    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:55.101377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.101262    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:55.101556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.101531    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:55.103100 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.103066    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:22:55.103814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.103785    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "kube-api-access-s74jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:55.104883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.104856    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:22:55.105001 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.104942    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token" pod "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" (UID: "7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:55.200746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200713    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200745    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200757    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200772    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200785    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200800    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200812    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200823    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74jz\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-kube-api-access-s74jz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.200982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.200835    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:22:55.911741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.911711    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"
Nov 02 08:22:55.912205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.911741    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6" event={"ID":"7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b","Type":"ContainerDied","Data":"1de861bfdca6d0640c73c38e4d8e39f72993815b42d93c8464401edd6f7e8adf"}
Nov 02 08:22:55.912205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:55.911802    4515 scope.go:122] "RemoveContainer" containerID="c37389f33e77f55bffbfff0dc1c8d25092d2d53eca4612832dd3ae671280013b"
Nov 02 08:22:56.005511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:56.005474    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:22:56.018022 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:56.017994    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e271b1-openshift-default-7c7d6f858c-kjmt6"]
Nov 02 08:22:57.375884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:57.375842    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" path="/var/lib/kubelet/pods/7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b/volumes"
Nov 02 08:22:58.014663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:58.014619    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing" containerID="cri-o://38afe07170b254ff0d2b3fc3db8a4aa74ed4e03b02fcbef61c3bf2ebe46d9b56" gracePeriod=25
Nov 02 08:22:58.951473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:22:58.951437    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:23:03.952604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:03.952560    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing" containerID="cri-o://30b247a145f3d5ad2b8bafcb2ed84db6cb70d5e274070096c2bb75a69f1ec1d1" gracePeriod=25
Nov 02 08:23:04.908095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:04.908053    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:07.833800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:07.833759    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:14.908423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:14.908379    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:17.724001 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:23:17.723960    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:23:17.833413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:17.833372    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:18.352283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.352256    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:23:18.439801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.439763    4515 scope.go:122] "RemoveContainer" containerID="38afe07170b254ff0d2b3fc3db8a4aa74ed4e03b02fcbef61c3bf2ebe46d9b56"
Nov 02 08:23:18.464091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.464066    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\" (UniqueName: \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\") pod \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") "
Nov 02 08:23:18.464385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.464367    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\") pod \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\" (UID: \"b163df41-7bca-4a0f-8e3f-5153e0aaa86e\") "
Nov 02 08:23:18.464828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.464768    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume" pod "b163df41-7bca-4a0f-8e3f-5153e0aaa86e" (UID: "b163df41-7bca-4a0f-8e3f-5153e0aaa86e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:18.465392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.465369    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:18.466483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.466457    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5" pod "b163df41-7bca-4a0f-8e3f-5153e0aaa86e" (UID: "b163df41-7bca-4a0f-8e3f-5153e0aaa86e"). InnerVolumeSpecName "kube-api-access-vcxh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:18.567488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:18.567404    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxh5\" (UniqueName: \"kubernetes.io/projected/b163df41-7bca-4a0f-8e3f-5153e0aaa86e-kube-api-access-vcxh5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:19.131053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.131023    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"
Nov 02 08:23:19.131053 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.131043    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" event={"ID":"b163df41-7bca-4a0f-8e3f-5153e0aaa86e","Type":"ContainerDied","Data":"38afe07170b254ff0d2b3fc3db8a4aa74ed4e03b02fcbef61c3bf2ebe46d9b56"}
Nov 02 08:23:19.131612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.131098    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw" event={"ID":"b163df41-7bca-4a0f-8e3f-5153e0aaa86e","Type":"ContainerDied","Data":"ad589099eb41125cf3213f2975b0f518b307da41ebeaa68a4a2c45db9b7ca265"}
Nov 02 08:23:19.214625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.214588    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:23:19.218401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.218374    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4489d-6745f7b5d7-h8gxw"]
Nov 02 08:23:19.376664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:19.376629    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" path="/var/lib/kubelet/pods/b163df41-7bca-4a0f-8e3f-5153e0aaa86e/volumes"
Nov 02 08:23:24.193258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.193225    4515 generic.go:356] "Generic (PLEG): container finished" podID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerID="30b247a145f3d5ad2b8bafcb2ed84db6cb70d5e274070096c2bb75a69f1ec1d1" exitCode=0
Nov 02 08:23:24.193805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.193265    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" event={"ID":"6072711d-30c3-48a8-b9d1-53c17aebe6a4","Type":"ContainerDied","Data":"30b247a145f3d5ad2b8bafcb2ed84db6cb70d5e274070096c2bb75a69f1ec1d1"}
Nov 02 08:23:24.245714 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.245686    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:23:24.365914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.365875    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\") pod \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") "
Nov 02 08:23:24.366134 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.365997    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\" (UniqueName: \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\") pod \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\" (UID: \"6072711d-30c3-48a8-b9d1-53c17aebe6a4\") "
Nov 02 08:23:24.366279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.366255    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume" pod "6072711d-30c3-48a8-b9d1-53c17aebe6a4" (UID: "6072711d-30c3-48a8-b9d1-53c17aebe6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:24.366545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.366526    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6072711d-30c3-48a8-b9d1-53c17aebe6a4-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:24.368117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.368093    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f" pod "6072711d-30c3-48a8-b9d1-53c17aebe6a4" (UID: "6072711d-30c3-48a8-b9d1-53c17aebe6a4"). InnerVolumeSpecName "kube-api-access-75j5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:24.467609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:24.467520    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75j5f\" (UniqueName: \"kubernetes.io/projected/6072711d-30c3-48a8-b9d1-53c17aebe6a4-kube-api-access-75j5f\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:25.203546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.203514    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx" event={"ID":"6072711d-30c3-48a8-b9d1-53c17aebe6a4","Type":"ContainerDied","Data":"2b018d11089cd2db6b1c41f97426feafc7fa6505b62087a51d434a67372fb761"}
Nov 02 08:23:25.203979 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.203573    4515 scope.go:122] "RemoveContainer" containerID="30b247a145f3d5ad2b8bafcb2ed84db6cb70d5e274070096c2bb75a69f1ec1d1"
Nov 02 08:23:25.203979 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.203520    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"
Nov 02 08:23:25.265062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.265030    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:23:25.268742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.268719    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f765b8-6cdcf47d95-jsgjx"]
Nov 02 08:23:25.385773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:25.385731    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" path="/var/lib/kubelet/pods/6072711d-30c3-48a8-b9d1-53c17aebe6a4/volumes"
Nov 02 08:23:36.309195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.309153    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:23:36.309705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.309468    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" podUID="772f51c2-91cb-4407-8cd9-ff54af72c83f" containerName="maas-api" containerID="cri-o://58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc" gracePeriod=30
Nov 02 08:23:36.627435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.627404    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:23:36.796399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.796362    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\") pod \"772f51c2-91cb-4407-8cd9-ff54af72c83f\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") "
Nov 02 08:23:36.796577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.796560    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\") pod \"772f51c2-91cb-4407-8cd9-ff54af72c83f\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") "
Nov 02 08:23:36.796638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.796631    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\" (UniqueName: \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\") pod \"772f51c2-91cb-4407-8cd9-ff54af72c83f\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") "
Nov 02 08:23:36.796710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.796671    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\") pod \"772f51c2-91cb-4407-8cd9-ff54af72c83f\" (UID: \"772f51c2-91cb-4407-8cd9-ff54af72c83f\") "
Nov 02 08:23:36.797114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.797079    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca" pod "772f51c2-91cb-4407-8cd9-ff54af72c83f" (UID: "772f51c2-91cb-4407-8cd9-ff54af72c83f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:36.797545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.797516    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772f51c2-91cb-4407-8cd9-ff54af72c83f-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:36.799077 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.799050    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs" pod "772f51c2-91cb-4407-8cd9-ff54af72c83f" (UID: "772f51c2-91cb-4407-8cd9-ff54af72c83f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:23:36.799201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.799129    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls" pod "772f51c2-91cb-4407-8cd9-ff54af72c83f" (UID: "772f51c2-91cb-4407-8cd9-ff54af72c83f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:23:36.799530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.799506    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87" pod "772f51c2-91cb-4407-8cd9-ff54af72c83f" (UID: "772f51c2-91cb-4407-8cd9-ff54af72c83f"). InnerVolumeSpecName "kube-api-access-zwg87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:36.901385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.900925    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:36.901385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.900965    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772f51c2-91cb-4407-8cd9-ff54af72c83f-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:36.901385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:36.900982    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwg87\" (UniqueName: \"kubernetes.io/projected/772f51c2-91cb-4407-8cd9-ff54af72c83f-kube-api-access-zwg87\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:37.323930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.323847    4515 generic.go:356] "Generic (PLEG): container finished" podID="772f51c2-91cb-4407-8cd9-ff54af72c83f" containerID="58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc" exitCode=0
Nov 02 08:23:37.323930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.323915    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" event={"ID":"772f51c2-91cb-4407-8cd9-ff54af72c83f","Type":"ContainerDied","Data":"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"}
Nov 02 08:23:37.324553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.324025    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z" event={"ID":"772f51c2-91cb-4407-8cd9-ff54af72c83f","Type":"ContainerDied","Data":"b6e3b8dab92b03dd5e26690af1466fb23d53325e9be675d1cc49ac28ba330ae6"}
Nov 02 08:23:37.324553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.324039    4515 scope.go:122] "RemoveContainer" containerID="58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"
Nov 02 08:23:37.324553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.323894    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"
Nov 02 08:23:37.339183 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.339149    4515 scope.go:122] "RemoveContainer" containerID="58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"
Nov 02 08:23:37.339647 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:23:37.339607    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc\": container with ID starting with 58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc not found: ID does not exist" containerID="58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"
Nov 02 08:23:37.339796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.339653    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc"} err="failed to get container status \"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc\": rpc error: code = NotFound desc = could not find container \"58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc\": container with ID starting with 58e1aba5ede79c9272bc6e5613084f360c5db249d4e89f692b952ab004abc0dc not found: ID does not exist"
Nov 02 08:23:37.386607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.386579    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:23:37.398374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:37.398348    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4489d-6dfbcb76df-4wp7z"]
Nov 02 08:23:39.377428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:39.377395    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772f51c2-91cb-4407-8cd9-ff54af72c83f" path="/var/lib/kubelet/pods/772f51c2-91cb-4407-8cd9-ff54af72c83f/volumes"
Nov 02 08:23:46.874260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:46.874221    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:23:46.907978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:46.907945    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:23:47.729031 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:23:47.728993    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:23:51.874662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:51.874615    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" podUID="55681672-5be5-4164-8d46-3be78e39c8ba" containerName="payload-processing" containerID="cri-o://f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f" gracePeriod=25
Nov 02 08:23:51.908471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:51.908412    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing" containerID="cri-o://5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28" gracePeriod=25
Nov 02 08:23:52.165223 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.165196    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:23:52.263204 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.263173    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\") pod \"55681672-5be5-4164-8d46-3be78e39c8ba\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") "
Nov 02 08:23:52.263465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.263366    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\" (UniqueName: \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\") pod \"55681672-5be5-4164-8d46-3be78e39c8ba\" (UID: \"55681672-5be5-4164-8d46-3be78e39c8ba\") "
Nov 02 08:23:52.263592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.263568    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume" pod "55681672-5be5-4164-8d46-3be78e39c8ba" (UID: "55681672-5be5-4164-8d46-3be78e39c8ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:52.263940 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.263912    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55681672-5be5-4164-8d46-3be78e39c8ba-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:52.265682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.265656    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz" pod "55681672-5be5-4164-8d46-3be78e39c8ba" (UID: "55681672-5be5-4164-8d46-3be78e39c8ba"). InnerVolumeSpecName "kube-api-access-sqjlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:52.365849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.365809    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjlz\" (UniqueName: \"kubernetes.io/projected/55681672-5be5-4164-8d46-3be78e39c8ba-kube-api-access-sqjlz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:23:52.436649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.436564    4515 generic.go:356] "Generic (PLEG): container finished" podID="55681672-5be5-4164-8d46-3be78e39c8ba" containerID="f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f" exitCode=0
Nov 02 08:23:52.436649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.436615    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"
Nov 02 08:23:52.436883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.436649    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" event={"ID":"55681672-5be5-4164-8d46-3be78e39c8ba","Type":"ContainerDied","Data":"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"}
Nov 02 08:23:52.436883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.436705    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m" event={"ID":"55681672-5be5-4164-8d46-3be78e39c8ba","Type":"ContainerDied","Data":"01a8027616ff68a8a5f09c8a6b6ee590b2e8158c888ff99590de8d968cdc97c8"}
Nov 02 08:23:52.436883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.436728    4515 scope.go:122] "RemoveContainer" containerID="f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"
Nov 02 08:23:52.448038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.448019    4515 scope.go:122] "RemoveContainer" containerID="f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"
Nov 02 08:23:52.448336 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:23:52.448297    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f\": container with ID starting with f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f not found: ID does not exist" containerID="f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"
Nov 02 08:23:52.448405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.448347    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f"} err="failed to get container status \"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f\": rpc error: code = NotFound desc = could not find container \"f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f\": container with ID starting with f8c1819dd0ce146a3b4b86a0accdcc2c7131da8bf39fa976a6df823d6216078f not found: ID does not exist"
Nov 02 08:23:52.491126 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.491093    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:23:52.494808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:52.494783    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dfd6f0-6b55b96879-6dp5m"]
Nov 02 08:23:53.375586 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:53.375553    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55681672-5be5-4164-8d46-3be78e39c8ba" path="/var/lib/kubelet/pods/55681672-5be5-4164-8d46-3be78e39c8ba/volumes"
Nov 02 08:23:56.254722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:23:56.254682    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:24:06.254543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:06.254506    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:24:09.297985 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:09.297952    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:24:12.230153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.230121    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:24:12.366272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.366235    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\") pod \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") "
Nov 02 08:24:12.366465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.366365    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\" (UniqueName: \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\") pod \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\" (UID: \"1094dfa4-e1c2-4d0c-814f-0ea0825a704d\") "
Nov 02 08:24:12.366949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.366919    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume" pod "1094dfa4-e1c2-4d0c-814f-0ea0825a704d" (UID: "1094dfa4-e1c2-4d0c-814f-0ea0825a704d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:24:12.367321 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.367285    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:24:12.369791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.369766    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4" pod "1094dfa4-e1c2-4d0c-814f-0ea0825a704d" (UID: "1094dfa4-e1c2-4d0c-814f-0ea0825a704d"). InnerVolumeSpecName "kube-api-access-rnrk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:24:12.469915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.469713    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnrk4\" (UniqueName: \"kubernetes.io/projected/1094dfa4-e1c2-4d0c-814f-0ea0825a704d-kube-api-access-rnrk4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:24:12.590066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.590029    4515 generic.go:356] "Generic (PLEG): container finished" podID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerID="5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28" exitCode=0
Nov 02 08:24:12.590291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.590082    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"
Nov 02 08:24:12.590291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.590101    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" event={"ID":"1094dfa4-e1c2-4d0c-814f-0ea0825a704d","Type":"ContainerDied","Data":"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"}
Nov 02 08:24:12.590291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.590178    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4" event={"ID":"1094dfa4-e1c2-4d0c-814f-0ea0825a704d","Type":"ContainerDied","Data":"529514075d23c5b79e8ecd7ac57667a9e6439ab386df04d7f5fbd04a0d59bd88"}
Nov 02 08:24:12.590291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.590207    4515 scope.go:122] "RemoveContainer" containerID="5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"
Nov 02 08:24:12.603143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.603118    4515 scope.go:122] "RemoveContainer" containerID="5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"
Nov 02 08:24:12.603532 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:24:12.603510    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28\": container with ID starting with 5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28 not found: ID does not exist" containerID="5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"
Nov 02 08:24:12.603613 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.603541    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28"} err="failed to get container status \"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28\": rpc error: code = NotFound desc = could not find container \"5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28\": container with ID starting with 5903081c1b37deb1e968b4be34b55d6d00df3c52363f32fd0d4e8c7fa1147e28 not found: ID does not exist"
Nov 02 08:24:12.650350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.650295    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:24:12.665059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:12.665015    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dfd6f0-998f7d5d7-vprg4"]
Nov 02 08:24:13.375170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:13.375114    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" path="/var/lib/kubelet/pods/1094dfa4-e1c2-4d0c-814f-0ea0825a704d/volumes"
Nov 02 08:24:17.725794 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:24:17.725757    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:24:41.201909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.201866    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:24:41.203242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203211    4515 cpu_manager.go:414] "containerMap: removing container" podUID="55681672-5be5-4164-8d46-3be78e39c8ba" containerName="payload-processing"
Nov 02 08:24:41.203242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203236    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55681672-5be5-4164-8d46-3be78e39c8ba" containerName="payload-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203251    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203258    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203266    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203271    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203286    4515 cpu_manager.go:414] "containerMap: removing container" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203291    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203331    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203337    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203344    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203349    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203357    4515 cpu_manager.go:414] "containerMap: removing container" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203362    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203369    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203373    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203382    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203387    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203404    4515 cpu_manager.go:414] "containerMap: removing container" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203409    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203427    4515 cpu_manager.go:414] "containerMap: removing container" podUID="772f51c2-91cb-4407-8cd9-ff54af72c83f" containerName="maas-api"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203431    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772f51c2-91cb-4407-8cd9-ff54af72c83f" containerName="maas-api"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203440    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy"
Nov 02 08:24:41.203452 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203444    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203560    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="756d4e3e-a850-49eb-b8ff-0bde2e052020" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203572    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b163df41-7bca-4a0f-8e3f-5153e0aaa86e" containerName="payload-pre-processing"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203578    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203583    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="189a9882-e40a-4271-bf60-01f888c4e043" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203588    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1094dfa4-e1c2-4d0c-814f-0ea0825a704d" containerName="payload-pre-processing"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203596    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="151f5506-8b72-459f-ba40-4479c93ca1c5" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203603    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="772f51c2-91cb-4407-8cd9-ff54af72c83f" containerName="maas-api"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203608    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b641edf-1f18-4f23-a30e-5d177cb1633e" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203616    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="55681672-5be5-4164-8d46-3be78e39c8ba" containerName="payload-processing"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203621    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94421cd-cc31-4922-95fd-06d462d95d38" containerName="istio-proxy"
Nov 02 08:24:41.204153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.203627    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6072711d-30c3-48a8-b9d1-53c17aebe6a4" containerName="payload-processing"
Nov 02 08:24:41.207648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.207629    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.220049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.220021    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:24:41.220919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.220902    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5be2cd-ab9c-4d7b-8c1c-ceec9caf782b" containerName="istio-proxy"
Nov 02 08:24:41.224616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.224596    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.234205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.234180    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:24:41.307409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307370    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.307621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307435    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hggf\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.307621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307602    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.307769 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307636    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.307769 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307741    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.307887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307788    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.307887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307846    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307947    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6l7t\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.307996    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308024    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308050    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308082    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.308330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308138    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.308330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308174    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308256    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308376    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.308523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308414    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.308523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.308440    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.322900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.322863    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:24:41.410074 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410033    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410110    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410148    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410168    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410190    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410243    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410275    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hggf\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410380    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410421    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410466    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410494    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410520    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410550    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410664    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l7t\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410710    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410735    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410759    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410789    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410804    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.410981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.410850    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.411485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411128    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.411485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411155    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.411485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411224    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.411485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411227    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.411485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411452    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.411689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411575    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.411757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411681    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.411843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.411823    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.413094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.413064    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.413232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.413153    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.413232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.413192    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.413578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.413561    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.437568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.437541    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hggf\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.437695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.437653    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.437695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.437653    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l7t\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hf7z5\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.437695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.437656    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-n97lr\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.524451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.524414    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:41.541392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.541361    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:41.849834 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:24:41.849799    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea689304_59b3_46c4_9be7_d12d25a78383.slice/crio-e86cdfdded4a7dc7f6cf3fb88a03d8dd8692dfa78cd5f404003d8bd7c11b8ac5 WatchSource:0}: Error finding container e86cdfdded4a7dc7f6cf3fb88a03d8dd8692dfa78cd5f404003d8bd7c11b8ac5: Status 404 returned error can't find the container with id e86cdfdded4a7dc7f6cf3fb88a03d8dd8692dfa78cd5f404003d8bd7c11b8ac5
Nov 02 08:24:41.851738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.851708    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:24:41.853001 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.852962    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.853102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.853032    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.853102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.853062    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.868509 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:24:41.868474    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded078af5_0ce7_4c07_9e6f_d4ca95a878d1.slice/crio-06a839bd7e847784493e79403fc8a7df37d308ed69419021f28aa4f87d1d5a6e WatchSource:0}: Error finding container 06a839bd7e847784493e79403fc8a7df37d308ed69419021f28aa4f87d1d5a6e: Status 404 returned error can't find the container with id 06a839bd7e847784493e79403fc8a7df37d308ed69419021f28aa4f87d1d5a6e
Nov 02 08:24:41.871012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.870991    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:24:41.872564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.872532    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.872627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.872600    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.872667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:41.872630    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:42.875999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.875955    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" event={"ID":"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1","Type":"ContainerStarted","Data":"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"}
Nov 02 08:24:42.876528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.876033    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" event={"ID":"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1","Type":"ContainerStarted","Data":"06a839bd7e847784493e79403fc8a7df37d308ed69419021f28aa4f87d1d5a6e"}
Nov 02 08:24:42.877968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.877945    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" event={"ID":"ea689304-59b3-46c4-9be7-d12d25a78383","Type":"ContainerStarted","Data":"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"}
Nov 02 08:24:42.878032 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.877994    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" event={"ID":"ea689304-59b3-46c4-9be7-d12d25a78383","Type":"ContainerStarted","Data":"e86cdfdded4a7dc7f6cf3fb88a03d8dd8692dfa78cd5f404003d8bd7c11b8ac5"}
Nov 02 08:24:42.962334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.953031    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podStartSLOduration=1.953009352 podStartE2EDuration="1.953009352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:42.938894042 +0000 UTC m=+3627.701330959" watchObservedRunningTime="2026-11-02 08:24:42.953009352 +0000 UTC m=+3627.715446259"
Nov 02 08:24:42.991763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:42.991700    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podStartSLOduration=1.9916861670000001 podStartE2EDuration="1.991686167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:42.99113268 +0000 UTC m=+3627.753569587" watchObservedRunningTime="2026-11-02 08:24:42.991686167 +0000 UTC m=+3627.754123074"
Nov 02 08:24:43.539345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.529679    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:43.545522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.545485    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:43.573568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.571250    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:43.573568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.571352    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:43.573568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.573437    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:43.573568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:43.573486    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:44.528334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:44.525664    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:44.528334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:44.525725    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:44.549333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:44.547749    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:44.549333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:44.547808    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:45.546333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:45.545761    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:45.546333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:45.545833    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:45.550334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:45.548054    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:45.550334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:45.548105    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:46.551333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:46.550468    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:46.551333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:46.550553    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:46.559432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:46.553567    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:46.559432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:46.553628    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:47.525330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:47.524695    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:47.525330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:47.524757    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:47.546327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:47.544752    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:47.546327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:47.544854    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:47.877854 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:24:47.877722    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:24:48.525262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:48.525222    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:48.525463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:48.525291    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:48.542521 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:48.542482    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:48.542708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:48.542551    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:49.525212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:49.525173    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hf7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Nov 02 08:24:49.525631 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:49.525253    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Nov 02 08:24:49.542002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:49.541970    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-n97lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 08:24:49.542151 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:49.542039    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 08:24:50.686966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.686935    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:50.687981 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.687949    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:50.689815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.689782    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:24:50.875730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.875699    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:50.877007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.876983    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:50.877596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:50.877573    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:24:56.285660 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.285627    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:24:56.290610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.290586    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.318436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.318399    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:24:56.336744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.336712    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.336952 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.336772    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.336950    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337042    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337093    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337255    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337413    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337480    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.337572 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.337554    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwxz\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.438791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.438758    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.438865    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.438900    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwxz\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.438954    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439020    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439097    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439167    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439230    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439294    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439397    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439437    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439706    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439828    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.439915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.439894    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.441606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.441584    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.441849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.441828    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.466426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.466392    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.466549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.466522    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwxz\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gppb5\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.609809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.609728    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:56.933014 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:24:56.932985    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd38826_5176_4d0c_ba84_d0797d6ed04c.slice/crio-d1c626fee18e69c9f4f3ac90e892895b86f587c31d163fd67561e9e439966680 WatchSource:0}: Error finding container d1c626fee18e69c9f4f3ac90e892895b86f587c31d163fd67561e9e439966680: Status 404 returned error can't find the container with id d1c626fee18e69c9f4f3ac90e892895b86f587c31d163fd67561e9e439966680
Nov 02 08:24:56.934921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.934861    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:24:56.936253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.936212    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:56.936390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.936299    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:56.936461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:56.936387    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:57.050831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:57.050798    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" event={"ID":"0dd38826-5176-4d0c-ba84-d0797d6ed04c","Type":"ContainerStarted","Data":"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"}
Nov 02 08:24:57.051032 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:57.050855    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" event={"ID":"0dd38826-5176-4d0c-ba84-d0797d6ed04c","Type":"ContainerStarted","Data":"d1c626fee18e69c9f4f3ac90e892895b86f587c31d163fd67561e9e439966680"}
Nov 02 08:24:57.099092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:57.097281    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podStartSLOduration=1.097263616 podStartE2EDuration="1.097263616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:57.095564038 +0000 UTC m=+3641.858000943" watchObservedRunningTime="2026-11-02 08:24:57.097263616 +0000 UTC m=+3641.859700521"
Nov 02 08:24:57.610367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:57.610256    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:24:58.612377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:58.612338    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:24:58.612891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:58.612401    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:24:59.610050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:59.610013    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:24:59.610252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:24:59.610074    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:00.099512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:00.099475    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:25:00.610790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:00.610754    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:25:00.610985 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:00.610823    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:01.610542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:01.610504    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:25:01.610982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:01.610568    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:02.610609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:02.610571    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:25:02.611028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:02.610631    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:03.610231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:03.610193    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:25:03.610440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:03.610263    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:04.610969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:04.610929    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gppb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 08:25:04.611417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:04.610996    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 08:25:05.802207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:05.802174    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:25:05.803530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:05.802753    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:25:05.803958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:05.803934    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:25:17.724593 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:25:17.724547    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:25:36.164017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:36.163983    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:25:39.151539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:39.151506    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:25:47.728657 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:25:47.728621    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:25:51.188725 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.188656    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" podUID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" containerName="main" containerID="cri-o://09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31" gracePeriod=45
Nov 02 08:25:51.475726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.475699    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:25:51.543728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.543690    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.543931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.543784    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.544069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544045    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.544267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544095    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:25:51.544367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544339    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.544367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544346    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:25:51.544495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544381    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.544495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544473    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\") pod \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\" (UID: \"1c5c4923-2d75-4e86-92bb-9f93a408acaf\") "
Nov 02 08:25:51.544667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.544642    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:25:51.545138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.545119    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.545274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.545255    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.545428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.545407    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.546643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.546614    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:25:51.546818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.546797    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "kube-api-access-jqgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:25:51.546922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.546908    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm" pod "1c5c4923-2d75-4e86-92bb-9f93a408acaf" (UID: "1c5c4923-2d75-4e86-92bb-9f93a408acaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:25:51.621550 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.621519    4515 generic.go:356] "Generic (PLEG): container finished" podID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" containerID="09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31" exitCode=0
Nov 02 08:25:51.621728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.621565    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"
Nov 02 08:25:51.621728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.621600    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" event={"ID":"1c5c4923-2d75-4e86-92bb-9f93a408acaf","Type":"ContainerDied","Data":"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"}
Nov 02 08:25:51.621728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.621647    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4" event={"ID":"1c5c4923-2d75-4e86-92bb-9f93a408acaf","Type":"ContainerDied","Data":"40160ce1cac24de2f7a7efae2b9daae1e00cf041fa87868709b09192f4774bec"}
Nov 02 08:25:51.621728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.621667    4515 scope.go:122] "RemoveContainer" containerID="09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"
Nov 02 08:25:51.633406 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.633385    4515 scope.go:122] "RemoveContainer" containerID="09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"
Nov 02 08:25:51.633689 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:25:51.633668    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31\": container with ID starting with 09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31 not found: ID does not exist" containerID="09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"
Nov 02 08:25:51.633771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.633697    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31"} err="failed to get container status \"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31\": rpc error: code = NotFound desc = could not find container \"09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31\": container with ID starting with 09f97529f09e32da526cb0455cc2a6781d4ead27ea723de5d9db889851affa31 not found: ID does not exist"
Nov 02 08:25:51.647721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.647694    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5c4923-2d75-4e86-92bb-9f93a408acaf-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.647868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.647727    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/1c5c4923-2d75-4e86-92bb-9f93a408acaf-kube-api-access-jqgql\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.647868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.647745    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5c4923-2d75-4e86-92bb-9f93a408acaf-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:25:51.675084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.675050    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:25:51.678812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:51.678786    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e6f4fe/test-model-e6f4fe-kserve-df77f5479-72cr4"]
Nov 02 08:25:53.376016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:25:53.375982    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" path="/var/lib/kubelet/pods/1c5c4923-2d75-4e86-92bb-9f93a408acaf/volumes"
Nov 02 08:26:01.304639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:01.304602    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:26:06.442898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:06.442855    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:26:06.503933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:06.503895    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:26:07.810952 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:07.810916    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:26:07.848377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:07.848343    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:26:11.443491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.443438    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" podUID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" containerName="payload-processing" containerID="cri-o://3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6" gracePeriod=25
Nov 02 08:26:11.504852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.504771    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing" containerID="cri-o://f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179" gracePeriod=25
Nov 02 08:26:11.796103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.796080    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:26:11.834541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.834502    4515 generic.go:356] "Generic (PLEG): container finished" podID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" containerID="3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6" exitCode=0
Nov 02 08:26:11.834751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.834548    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"
Nov 02 08:26:11.834751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.834587    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" event={"ID":"f8e6cce6-585b-4c66-bcf1-ee881a59052b","Type":"ContainerDied","Data":"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"}
Nov 02 08:26:11.834751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.834642    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7" event={"ID":"f8e6cce6-585b-4c66-bcf1-ee881a59052b","Type":"ContainerDied","Data":"bc2d4cee1c24d02eb742d45b2ad0caf09b091a50b03a3e5152fe78497c8e0261"}
Nov 02 08:26:11.834751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.834671    4515 scope.go:122] "RemoveContainer" containerID="3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"
Nov 02 08:26:11.849119 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.849096    4515 scope.go:122] "RemoveContainer" containerID="3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"
Nov 02 08:26:11.849450 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:11.849431    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6\": container with ID starting with 3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6 not found: ID does not exist" containerID="3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"
Nov 02 08:26:11.849517 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.849461    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6"} err="failed to get container status \"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6\": rpc error: code = NotFound desc = could not find container \"3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6\": container with ID starting with 3cb267e67371fff75051e58fcbec0cffbb63f246fd93cd51d7c2b1c973b0b2e6 not found: ID does not exist"
Nov 02 08:26:11.954487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.954450    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\" (UniqueName: \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\") pod \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") "
Nov 02 08:26:11.954824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.954784    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\") pod \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\" (UID: \"f8e6cce6-585b-4c66-bcf1-ee881a59052b\") "
Nov 02 08:26:11.955220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.955195    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume" pod "f8e6cce6-585b-4c66-bcf1-ee881a59052b" (UID: "f8e6cce6-585b-4c66-bcf1-ee881a59052b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:11.955433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.955418    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e6cce6-585b-4c66-bcf1-ee881a59052b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:11.956771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:11.956742    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp" pod "f8e6cce6-585b-4c66-bcf1-ee881a59052b" (UID: "f8e6cce6-585b-4c66-bcf1-ee881a59052b"). InnerVolumeSpecName "kube-api-access-rhdvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:12.056447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.056365    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdvp\" (UniqueName: \"kubernetes.io/projected/f8e6cce6-585b-4c66-bcf1-ee881a59052b-kube-api-access-rhdvp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:12.182489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.182451    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:26:12.186796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.186766    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6f4fe-785d685d77-w4vw7"]
Nov 02 08:26:12.307213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.307124    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:12.811453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.811409    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" podUID="48a3a97f-9032-4471-8867-96590684a65e" containerName="payload-processing" containerID="cri-o://09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422" gracePeriod=25
Nov 02 08:26:12.849574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:12.849537    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing" containerID="cri-o://687ea352bac1ba44e5483ef39f50d99f6e72d4f7ba9c5327e5bb8d5518fd0514" gracePeriod=25
Nov 02 08:26:13.098244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.098213    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:26:13.274522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.274488    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\" (UniqueName: \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\") pod \"48a3a97f-9032-4471-8867-96590684a65e\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") "
Nov 02 08:26:13.274774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.274760    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\") pod \"48a3a97f-9032-4471-8867-96590684a65e\" (UID: \"48a3a97f-9032-4471-8867-96590684a65e\") "
Nov 02 08:26:13.275154 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.275129    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume" pod "48a3a97f-9032-4471-8867-96590684a65e" (UID: "48a3a97f-9032-4471-8867-96590684a65e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:13.275296 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.275280    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a3a97f-9032-4471-8867-96590684a65e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:13.276663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.276639    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz" pod "48a3a97f-9032-4471-8867-96590684a65e" (UID: "48a3a97f-9032-4471-8867-96590684a65e"). InnerVolumeSpecName "kube-api-access-jqjxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:13.375458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.375371    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" path="/var/lib/kubelet/pods/f8e6cce6-585b-4c66-bcf1-ee881a59052b/volumes"
Nov 02 08:26:13.376232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.376206    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjxz\" (UniqueName: \"kubernetes.io/projected/48a3a97f-9032-4471-8867-96590684a65e-kube-api-access-jqjxz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:13.854061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.854027    4515 generic.go:356] "Generic (PLEG): container finished" podID="48a3a97f-9032-4471-8867-96590684a65e" containerID="09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422" exitCode=0
Nov 02 08:26:13.854542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.854074    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"
Nov 02 08:26:13.854542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.854111    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" event={"ID":"48a3a97f-9032-4471-8867-96590684a65e","Type":"ContainerDied","Data":"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"}
Nov 02 08:26:13.854542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.854174    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs" event={"ID":"48a3a97f-9032-4471-8867-96590684a65e","Type":"ContainerDied","Data":"cc3d65837cebcc863cd743e3fd7001d892fb7beb2fa848d387a625906487dc61"}
Nov 02 08:26:13.854542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.854202    4515 scope.go:122] "RemoveContainer" containerID="09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"
Nov 02 08:26:13.864578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.864560    4515 scope.go:122] "RemoveContainer" containerID="09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"
Nov 02 08:26:13.864857 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:13.864840    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422\": container with ID starting with 09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422 not found: ID does not exist" containerID="09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"
Nov 02 08:26:13.864934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.864867    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422"} err="failed to get container status \"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422\": rpc error: code = NotFound desc = could not find container \"09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422\": container with ID starting with 09cbc4e235cf61e9d5ff0a33aa51ffd7a7512192690d98a42ab103f40e9a2422 not found: ID does not exist"
Nov 02 08:26:13.908916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.908878    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:26:13.912696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:13.912668    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78ddbc-78c556dbb9-mw6gs"]
Nov 02 08:26:15.375895 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:15.375859    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a3a97f-9032-4471-8867-96590684a65e" path="/var/lib/kubelet/pods/48a3a97f-9032-4471-8867-96590684a65e/volumes"
Nov 02 08:26:16.330270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.330231    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" podUID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" containerName="main" containerID="cri-o://f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f" gracePeriod=45
Nov 02 08:26:16.633541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.633514    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:26:16.720105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720064    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720156    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720268    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720379    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720436    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720505    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\" (UniqueName: \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\") pod \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\" (UID: \"7ac7b209-3d01-49a5-9961-4a2dbcc799b8\") "
Nov 02 08:26:16.720602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720528    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:16.720716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720591    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:16.720716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.720621    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:16.721086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.721061    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.721086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.721084    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.721266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.721105    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.723279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.723254    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:16.723524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.723504    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "kube-api-access-knddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:16.724115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.724098    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs" pod "7ac7b209-3d01-49a5-9961-4a2dbcc799b8" (UID: "7ac7b209-3d01-49a5-9961-4a2dbcc799b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:16.823351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.823272    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.823351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.823354    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knddg\" (UniqueName: \"kubernetes.io/projected/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-kube-api-access-knddg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.823654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.823381    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7b209-3d01-49a5-9961-4a2dbcc799b8-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:16.878050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.877958    4515 generic.go:356] "Generic (PLEG): container finished" podID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" containerID="f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f" exitCode=0
Nov 02 08:26:16.878246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.878095    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"
Nov 02 08:26:16.878246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.878102    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" event={"ID":"7ac7b209-3d01-49a5-9961-4a2dbcc799b8","Type":"ContainerDied","Data":"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"}
Nov 02 08:26:16.878246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.878225    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb" event={"ID":"7ac7b209-3d01-49a5-9961-4a2dbcc799b8","Type":"ContainerDied","Data":"34135273abea659126a7713b6325a86b7d04a405aeca6b52362b15f58dd4b916"}
Nov 02 08:26:16.878465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.878250    4515 scope.go:122] "RemoveContainer" containerID="f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"
Nov 02 08:26:16.890060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.890042    4515 scope.go:122] "RemoveContainer" containerID="f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"
Nov 02 08:26:16.890375 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:16.890346    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f\": container with ID starting with f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f not found: ID does not exist" containerID="f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"
Nov 02 08:26:16.890492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.890381    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f"} err="failed to get container status \"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f\": rpc error: code = NotFound desc = could not find container \"f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f\": container with ID starting with f55798b46f40c387649e0fd13cda7e34a689ebf994dcb7a6885a5748c713206f not found: ID does not exist"
Nov 02 08:26:16.937794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.937753    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:26:16.941861 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:16.941834    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d042a8/test-model-d042a8-kserve-b68d88c7-7mdrb"]
Nov 02 08:26:17.375474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:17.375442    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" path="/var/lib/kubelet/pods/7ac7b209-3d01-49a5-9961-4a2dbcc799b8/volumes"
Nov 02 08:26:17.725251 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:17.725164    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:26:19.796728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:19.796685    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:22.307007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:22.306968    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:23.781492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:23.781458    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:26:23.781955 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:23.781680    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" podUID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" containerName="maas-api" containerID="cri-o://1ee74ab5a89043c82524ed9fc6fbacdd793604f1f8c79fe1e62ec9a41e5fb3cf" gracePeriod=30
Nov 02 08:26:23.934599 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:23.934567    4515 generic.go:356] "Generic (PLEG): container finished" podID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" containerID="1ee74ab5a89043c82524ed9fc6fbacdd793604f1f8c79fe1e62ec9a41e5fb3cf" exitCode=0
Nov 02 08:26:23.934773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:23.934647    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" event={"ID":"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3","Type":"ContainerDied","Data":"1ee74ab5a89043c82524ed9fc6fbacdd793604f1f8c79fe1e62ec9a41e5fb3cf"}
Nov 02 08:26:24.313283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.313248    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:26:24.413089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.412988    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\") pod \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") "
Nov 02 08:26:24.413089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.413077    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\" (UniqueName: \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\") pod \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") "
Nov 02 08:26:24.413420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.413190    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\") pod \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") "
Nov 02 08:26:24.413420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.413236    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\") pod \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\" (UID: \"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3\") "
Nov 02 08:26:24.414391 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.414357    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca" pod "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" (UID: "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:24.415893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.415865    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls" pod "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" (UID: "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:24.416227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.416194    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs" pod "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" (UID: "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:24.416376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.416348    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp" pod "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" (UID: "2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3"). InnerVolumeSpecName "kube-api-access-wndwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:24.515138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.515096    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.515138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.515131    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.515138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.515141    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndwp\" (UniqueName: \"kubernetes.io/projected/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-kube-api-access-wndwp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.515438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.515150    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.943434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.943406    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"
Nov 02 08:26:24.943942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.943444    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j" event={"ID":"2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3","Type":"ContainerDied","Data":"5348cf3b9bdc5f673aa2018992487b568006c6fc7dadf559db798d3435387d50"}
Nov 02 08:26:24.943942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:24.943518    4515 scope.go:122] "RemoveContainer" containerID="1ee74ab5a89043c82524ed9fc6fbacdd793604f1f8c79fe1e62ec9a41e5fb3cf"
Nov 02 08:26:25.003554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:25.003521    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:26:25.015853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:25.015820    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78ddbc-67c8788899-7db4j"]
Nov 02 08:26:25.389022 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:25.388978    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" path="/var/lib/kubelet/pods/2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3/volumes"
Nov 02 08:26:29.796989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:29.796874    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:31.665932 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.665869    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:26:31.666753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.666708    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" containerID="cri-o://7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695" gracePeriod=30
Nov 02 08:26:31.988868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.988833    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:26:31.997365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.997290    4515 generic.go:356] "Generic (PLEG): container finished" podID="221d40e4-336f-46c8-b6b8-466427f600fc" containerID="f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179" exitCode=0
Nov 02 08:26:31.997365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.997360    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" event={"ID":"221d40e4-336f-46c8-b6b8-466427f600fc","Type":"ContainerDied","Data":"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"}
Nov 02 08:26:31.997618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.997342    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"
Nov 02 08:26:31.997618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.997417    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx" event={"ID":"221d40e4-336f-46c8-b6b8-466427f600fc","Type":"ContainerDied","Data":"426f51dde5173357fc1e7bc0ed6eb240009cbb1099c2ab66b28d5e3a35ab50a1"}
Nov 02 08:26:31.997618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:31.997444    4515 scope.go:122] "RemoveContainer" containerID="f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"
Nov 02 08:26:32.016418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.016396    4515 scope.go:122] "RemoveContainer" containerID="f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"
Nov 02 08:26:32.016737 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:32.016713    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179\": container with ID starting with f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179 not found: ID does not exist" containerID="f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"
Nov 02 08:26:32.016817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.016746    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179"} err="failed to get container status \"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179\": rpc error: code = NotFound desc = could not find container \"f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179\": container with ID starting with f5da0745d46a747555f9fff94ab8990ab9b5ca4b51f609e2af0a0a0d4efac179 not found: ID does not exist"
Nov 02 08:26:32.048187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.048147    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\") pod \"221d40e4-336f-46c8-b6b8-466427f600fc\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") "
Nov 02 08:26:32.048382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.048344    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\" (UniqueName: \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\") pod \"221d40e4-336f-46c8-b6b8-466427f600fc\" (UID: \"221d40e4-336f-46c8-b6b8-466427f600fc\") "
Nov 02 08:26:32.048595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.048570    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume" pod "221d40e4-336f-46c8-b6b8-466427f600fc" (UID: "221d40e4-336f-46c8-b6b8-466427f600fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:32.049188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.049165    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d40e4-336f-46c8-b6b8-466427f600fc-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:32.051162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.051132    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt" pod "221d40e4-336f-46c8-b6b8-466427f600fc" (UID: "221d40e4-336f-46c8-b6b8-466427f600fc"). InnerVolumeSpecName "kube-api-access-4mvnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:32.151123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.151081    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvnt\" (UniqueName: \"kubernetes.io/projected/221d40e4-336f-46c8-b6b8-466427f600fc-kube-api-access-4mvnt\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:32.345628 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.345591    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:26:32.358037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.358000    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6f4fe-65b98ddccf-sxwhx"]
Nov 02 08:26:32.381322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.381272    4515 patch_prober.go:29] interesting pod/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:26:32.381495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:32.381370    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:26:33.008727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.008686    4515 generic.go:356] "Generic (PLEG): container finished" podID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerID="687ea352bac1ba44e5483ef39f50d99f6e72d4f7ba9c5327e5bb8d5518fd0514" exitCode=0
Nov 02 08:26:33.009247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.008754    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" event={"ID":"672d5549-1205-4fb8-a1e6-56a2971e41c9","Type":"ContainerDied","Data":"687ea352bac1ba44e5483ef39f50d99f6e72d4f7ba9c5327e5bb8d5518fd0514"}
Nov 02 08:26:33.149843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.149808    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:26:33.165121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.165082    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\") pod \"672d5549-1205-4fb8-a1e6-56a2971e41c9\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") "
Nov 02 08:26:33.165405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.165380    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\" (UniqueName: \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\") pod \"672d5549-1205-4fb8-a1e6-56a2971e41c9\" (UID: \"672d5549-1205-4fb8-a1e6-56a2971e41c9\") "
Nov 02 08:26:33.165568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.165540    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume" pod "672d5549-1205-4fb8-a1e6-56a2971e41c9" (UID: "672d5549-1205-4fb8-a1e6-56a2971e41c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:33.165977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.165959    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672d5549-1205-4fb8-a1e6-56a2971e41c9-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:33.167473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.167451    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b" pod "672d5549-1205-4fb8-a1e6-56a2971e41c9" (UID: "672d5549-1205-4fb8-a1e6-56a2971e41c9"). InnerVolumeSpecName "kube-api-access-sht7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:33.267835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.267797    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sht7b\" (UniqueName: \"kubernetes.io/projected/672d5549-1205-4fb8-a1e6-56a2971e41c9-kube-api-access-sht7b\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:33.376217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:33.376138    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" path="/var/lib/kubelet/pods/221d40e4-336f-46c8-b6b8-466427f600fc/volumes"
Nov 02 08:26:34.029169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:34.029136    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"
Nov 02 08:26:34.029169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:34.029157    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b" event={"ID":"672d5549-1205-4fb8-a1e6-56a2971e41c9","Type":"ContainerDied","Data":"90d618b58abfe062376f0a155c7bdef09d4243ef92adb48ebc7161cb655f6cfd"}
Nov 02 08:26:34.029786 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:34.029254    4515 scope.go:122] "RemoveContainer" containerID="687ea352bac1ba44e5483ef39f50d99f6e72d4f7ba9c5327e5bb8d5518fd0514"
Nov 02 08:26:34.082966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:34.082929    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:26:34.095661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:34.095626    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78ddbc-db7449c5-v848b"]
Nov 02 08:26:35.378047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:35.378014    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" path="/var/lib/kubelet/pods/672d5549-1205-4fb8-a1e6-56a2971e41c9/volumes"
Nov 02 08:26:36.547349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.547299    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 08:26:36.547794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.547500    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" podUID="0c34e82d-1916-4a64-b82c-39f022d509a4" containerName="discovery" containerID="cri-o://720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8" gracePeriod=30
Nov 02 08:26:36.868722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.868680    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 08:26:36.984302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984196    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984361    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984465    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984502    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984575    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984710    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.984783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.984744    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\") pod \"0c34e82d-1916-4a64-b82c-39f022d509a4\" (UID: \"0c34e82d-1916-4a64-b82c-39f022d509a4\") "
Nov 02 08:26:36.985812 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.985779    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:36.986887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.986856    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:36.987907 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.987872    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:36.988040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.987977    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:36.988253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.988234    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:36.989271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.989244    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:36.989442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.989382    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq" pod "0c34e82d-1916-4a64-b82c-39f022d509a4" (UID: "0c34e82d-1916-4a64-b82c-39f022d509a4"). InnerVolumeSpecName "kube-api-access-z5ldq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:36.991163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:36.991139    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:26:37.086283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086185    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086409    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086552 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086484    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086570    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086600    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086634    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086724    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086813    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.086892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.086851    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\") pod \"1252f400-2f23-47b6-97c2-5e564b1725ff\" (UID: \"1252f400-2f23-47b6-97c2-5e564b1725ff\") "
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087546    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087557    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087613    4515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-kubeconfig\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087629    4515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-cacerts\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087644    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-ca-configmap\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087658    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ldq\" (UniqueName: \"kubernetes.io/projected/0c34e82d-1916-4a64-b82c-39f022d509a4-kube-api-access-z5ldq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.087670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087672    4515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c34e82d-1916-4a64-b82c-39f022d509a4-local-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.088180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087685    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c34e82d-1916-4a64-b82c-39f022d509a4-istio-csr-dns-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.088180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.087800    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:37.088180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.088067    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:37.088412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.088365    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:37.088412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.088378    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:37.089274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.089178    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:26:37.090828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.090474    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:37.091673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.091645    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:37.102182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.102146    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg" pod "1252f400-2f23-47b6-97c2-5e564b1725ff" (UID: "1252f400-2f23-47b6-97c2-5e564b1725ff"). InnerVolumeSpecName "kube-api-access-zp6vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:37.184638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.184596    4515 generic.go:356] "Generic (PLEG): container finished" podID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerID="7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695" exitCode=0
Nov 02 08:26:37.184847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.184635    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerDied","Data":"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"}
Nov 02 08:26:37.184847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.184688    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"
Nov 02 08:26:37.184847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.184704    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj" event={"ID":"1252f400-2f23-47b6-97c2-5e564b1725ff","Type":"ContainerDied","Data":"f33282888ae63ba5aae54e051310036afd0f35aeaa5deb7bae0abc515447b661"}
Nov 02 08:26:37.184847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.184736    4515 scope.go:122] "RemoveContainer" containerID="7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"
Nov 02 08:26:37.186718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.186692    4515 generic.go:356] "Generic (PLEG): container finished" podID="0c34e82d-1916-4a64-b82c-39f022d509a4" containerID="720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8" exitCode=0
Nov 02 08:26:37.186810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.186761    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" event={"ID":"0c34e82d-1916-4a64-b82c-39f022d509a4","Type":"ContainerDied","Data":"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"}
Nov 02 08:26:37.186810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.186804    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk" event={"ID":"0c34e82d-1916-4a64-b82c-39f022d509a4","Type":"ContainerDied","Data":"7b0418496dbd02c1759f8c21a74748a2d62f08e4d76458d50364150edc4e31e6"}
Nov 02 08:26:37.186899 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.186834    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"
Nov 02 08:26:37.188464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188443    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188468    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188482    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188495    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188508    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188520    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1252f400-2f23-47b6-97c2-5e564b1725ff-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188533    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6vg\" (UniqueName: \"kubernetes.io/projected/1252f400-2f23-47b6-97c2-5e564b1725ff-kube-api-access-zp6vg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188549    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.188561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.188562    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1252f400-2f23-47b6-97c2-5e564b1725ff-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:37.196532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.196451    4515 scope.go:122] "RemoveContainer" containerID="85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"
Nov 02 08:26:37.207374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.207351    4515 scope.go:122] "RemoveContainer" containerID="7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"
Nov 02 08:26:37.207661 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:37.207642    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695\": container with ID starting with 7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695 not found: ID does not exist" containerID="7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"
Nov 02 08:26:37.207715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.207672    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695"} err="failed to get container status \"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695\": rpc error: code = NotFound desc = could not find container \"7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695\": container with ID starting with 7582d5c52c6485ccfcf4ec0beda3f36b4722c49626061230abfa3eacc9d99695 not found: ID does not exist"
Nov 02 08:26:37.207715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.207693    4515 scope.go:122] "RemoveContainer" containerID="85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"
Nov 02 08:26:37.207934 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:37.207917    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47\": container with ID starting with 85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47 not found: ID does not exist" containerID="85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"
Nov 02 08:26:37.207977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.207938    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47"} err="failed to get container status \"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47\": rpc error: code = NotFound desc = could not find container \"85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47\": container with ID starting with 85bf9130c658de88a15dda5bf03d2b70f8365adc39d5cd1f93012ba1a4516d47 not found: ID does not exist"
Nov 02 08:26:37.207977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.207951    4515 scope.go:122] "RemoveContainer" containerID="720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"
Nov 02 08:26:37.228653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.228631    4515 scope.go:122] "RemoveContainer" containerID="720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"
Nov 02 08:26:37.229038 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:37.229017    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8\": container with ID starting with 720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8 not found: ID does not exist" containerID="720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"
Nov 02 08:26:37.229108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.229047    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8"} err="failed to get container status \"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8\": rpc error: code = NotFound desc = could not find container \"720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8\": container with ID starting with 720505243f8b3346aaaa9bc2e4b57df2ae5b9fb38cf8196d1eb7cf5fc3fa51d8 not found: ID does not exist"
Nov 02 08:26:37.273133 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.273087    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 08:26:37.312031 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.311994    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z89sk"]
Nov 02 08:26:37.376987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.376903    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c34e82d-1916-4a64-b82c-39f022d509a4" path="/var/lib/kubelet/pods/0c34e82d-1916-4a64-b82c-39f022d509a4/volumes"
Nov 02 08:26:37.377348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.377330    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:26:37.378711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.378693    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-78ddbc-openshift-default-668df4d547-q8gsj"]
Nov 02 08:26:37.501023 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:37.500989    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:26:39.376957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:39.376925    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" path="/var/lib/kubelet/pods/1252f400-2f23-47b6-97c2-5e564b1725ff/volumes"
Nov 02 08:26:39.716007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:39.715923    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:26:42.502301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.502260    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" podUID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" containerName="payload-processing" containerID="cri-o://0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235" gracePeriod=25
Nov 02 08:26:42.809251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.809222    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:26:42.885277 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.885225    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\") pod \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") "
Nov 02 08:26:42.885489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.885374    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\" (UniqueName: \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\") pod \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\" (UID: \"b9d05e55-eb9e-4032-89bd-32e65237f7eb\") "
Nov 02 08:26:42.885719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.885683    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume" pod "b9d05e55-eb9e-4032-89bd-32e65237f7eb" (UID: "b9d05e55-eb9e-4032-89bd-32e65237f7eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:42.886174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.886147    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d05e55-eb9e-4032-89bd-32e65237f7eb-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:42.887891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.887844    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74" pod "b9d05e55-eb9e-4032-89bd-32e65237f7eb" (UID: "b9d05e55-eb9e-4032-89bd-32e65237f7eb"). InnerVolumeSpecName "kube-api-access-2rx74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:42.987980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:42.987929    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rx74\" (UniqueName: \"kubernetes.io/projected/b9d05e55-eb9e-4032-89bd-32e65237f7eb-kube-api-access-2rx74\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:26:43.259770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.259726    4515 generic.go:356] "Generic (PLEG): container finished" podID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" containerID="0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235" exitCode=0
Nov 02 08:26:43.259947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.259798    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" event={"ID":"b9d05e55-eb9e-4032-89bd-32e65237f7eb","Type":"ContainerDied","Data":"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"}
Nov 02 08:26:43.259947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.259893    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8" event={"ID":"b9d05e55-eb9e-4032-89bd-32e65237f7eb","Type":"ContainerDied","Data":"13772a48a9ff1be816c0083979ebff8c3eda22ae50f2b60b8548ff345e9c3cff"}
Nov 02 08:26:43.259947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.259807    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"
Nov 02 08:26:43.259947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.259917    4515 scope.go:122] "RemoveContainer" containerID="0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"
Nov 02 08:26:43.274257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.274233    4515 scope.go:122] "RemoveContainer" containerID="0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"
Nov 02 08:26:43.274829 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:43.274804    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235\": container with ID starting with 0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235 not found: ID does not exist" containerID="0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"
Nov 02 08:26:43.274977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.274841    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235"} err="failed to get container status \"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235\": rpc error: code = NotFound desc = could not find container \"0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235\": container with ID starting with 0e2e34d054f813511787c8b9b8ddcf9eaac4df2c62d316a7c45a1c645a7d5235 not found: ID does not exist"
Nov 02 08:26:43.314861 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.314819    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:26:43.319202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.319173    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d042a8-7b4d8869f7-v6fw8"]
Nov 02 08:26:43.375516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:43.375482    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" path="/var/lib/kubelet/pods/b9d05e55-eb9e-4032-89bd-32e65237f7eb/volumes"
Nov 02 08:26:45.816722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:45.816687    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:26:45.834843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:45.834812    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:26:47.724806 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:26:47.724766    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:26:50.818012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:50.817947    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing" containerID="cri-o://8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8" gracePeriod=25
Nov 02 08:26:50.835828 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:50.835766    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing" containerID="cri-o://97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f" gracePeriod=25
Nov 02 08:26:57.000672 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:57.000634    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:57.000672 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:26:57.000667    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:06.652814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.652775    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:27:06.653569 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653553    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653571    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653584    4515 cpu_manager.go:414] "containerMap: removing container" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653590    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653605    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" containerName="maas-api"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653610    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" containerName="maas-api"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653620    4515 cpu_manager.go:414] "containerMap: removing container" podUID="48a3a97f-9032-4471-8867-96590684a65e" containerName="payload-processing"
Nov 02 08:27:06.653632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653625    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a3a97f-9032-4471-8867-96590684a65e" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653650    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653655    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653669    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0c34e82d-1916-4a64-b82c-39f022d509a4" containerName="discovery"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653673    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c34e82d-1916-4a64-b82c-39f022d509a4" containerName="discovery"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653681    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" containerName="main"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653686    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" containerName="main"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653699    4515 cpu_manager.go:414] "containerMap: removing container" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653704    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653712    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" containerName="main"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653716    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" containerName="main"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653728    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653733    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653832    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c49db03-b0bc-41e8-92fc-8d9fdec4d6b3" containerName="maas-api"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653843    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a3a97f-9032-4471-8867-96590684a65e" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653850    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653858    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d05e55-eb9e-4032-89bd-32e65237f7eb" containerName="payload-processing"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653863    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c34e82d-1916-4a64-b82c-39f022d509a4" containerName="discovery"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653868    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5c4923-2d75-4e86-92bb-9f93a408acaf" containerName="main"
Nov 02 08:27:06.653868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653874    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="221d40e4-336f-46c8-b6b8-466427f600fc" containerName="payload-pre-processing"
Nov 02 08:27:06.654590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653880    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac7b209-3d01-49a5-9961-4a2dbcc799b8" containerName="main"
Nov 02 08:27:06.654590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653886    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e6cce6-585b-4c66-bcf1-ee881a59052b" containerName="payload-processing"
Nov 02 08:27:06.654590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.653892    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="672d5549-1205-4fb8-a1e6-56a2971e41c9" containerName="payload-pre-processing"
Nov 02 08:27:06.657925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.657904    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.688454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.688417    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 08:27:06.688647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.688495    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 08:27:06.688647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.688520    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bq7lr\""
Nov 02 08:27:06.688647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.688541    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 08:27:06.694697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.694665    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:27:06.730179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730140    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730242    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730395    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730448    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz22k\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730477    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730514    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.730650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.730597    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832269    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832356    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832453    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832499    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz22k\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832529    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832555    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.832690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.832603    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.833272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.833244    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.834978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.834946    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.835088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.835007    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.835254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.835237    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.835415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.835392    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.856024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.855989    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.856398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.856372    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz22k\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmcv2\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:06.971775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:06.971684    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:07.000301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.000260    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:07.000474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.000378    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:07.295365 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:27:07.295332    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffbfbc9_cdb1_4160_a90a_6a4b3936d81f.slice/crio-746f0b55c26d4c2c7c0f87bcb167a3176932f530a295f8994a44e381020fcb45 WatchSource:0}: Error finding container 746f0b55c26d4c2c7c0f87bcb167a3176932f530a295f8994a44e381020fcb45: Status 404 returned error can't find the container with id 746f0b55c26d4c2c7c0f87bcb167a3176932f530a295f8994a44e381020fcb45
Nov 02 08:27:07.296348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.296320    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:27:07.298295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.298261    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:27:07.298423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.298343    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:27:07.447762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.447718    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" event={"ID":"effbfbc9-cdb1-4160-a90a-6a4b3936d81f","Type":"ContainerStarted","Data":"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"}
Nov 02 08:27:07.447957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.447794    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:07.447957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.447816    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" event={"ID":"effbfbc9-cdb1-4160-a90a-6a4b3936d81f","Type":"ContainerStarted","Data":"746f0b55c26d4c2c7c0f87bcb167a3176932f530a295f8994a44e381020fcb45"}
Nov 02 08:27:07.501143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:07.501071    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" podStartSLOduration=1.5010585239999998 podStartE2EDuration="1.501058524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:07.5000195 +0000 UTC m=+3772.262456406" watchObservedRunningTime="2026-11-02 08:27:07.501058524 +0000 UTC m=+3772.263495430"
Nov 02 08:27:08.458033 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:08.457994    4515 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kmcv2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:27:08.458558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:08.458047    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:27:11.124735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.124700    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:27:11.191887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.191861    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:27:11.202756 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.202729    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\" (UniqueName: \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\") pod \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") "
Nov 02 08:27:11.202936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.202839    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\") pod \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\" (UID: \"ac1292b7-fafb-42ce-b43c-fe82b356ca0b\") "
Nov 02 08:27:11.203220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.203197    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume" pod "ac1292b7-fafb-42ce-b43c-fe82b356ca0b" (UID: "ac1292b7-fafb-42ce-b43c-fe82b356ca0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:11.203378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.203362    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:11.204922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.204899    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc" pod "ac1292b7-fafb-42ce-b43c-fe82b356ca0b" (UID: "ac1292b7-fafb-42ce-b43c-fe82b356ca0b"). InnerVolumeSpecName "kube-api-access-zpqhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:11.304148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.304112    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\") pod \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") "
Nov 02 08:27:11.304347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.304300    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\" (UniqueName: \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\") pod \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\" (UID: \"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a\") "
Nov 02 08:27:11.304608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.304582    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume" pod "02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" (UID: "02e1e52e-7379-4863-b1e3-3dfa3d1fc39a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:11.304836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.304822    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqhc\" (UniqueName: \"kubernetes.io/projected/ac1292b7-fafb-42ce-b43c-fe82b356ca0b-kube-api-access-zpqhc\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:11.304875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.304839    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:11.306509 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.306479    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz" pod "02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" (UID: "02e1e52e-7379-4863-b1e3-3dfa3d1fc39a"). InnerVolumeSpecName "kube-api-access-74dbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:11.406528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.406484    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74dbz\" (UniqueName: \"kubernetes.io/projected/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a-kube-api-access-74dbz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:11.457580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.457535    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:27:11.480494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.480456    4515 generic.go:356] "Generic (PLEG): container finished" podID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerID="8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8" exitCode=0
Nov 02 08:27:11.480780 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.480761    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"
Nov 02 08:27:11.481535 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.481501    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" event={"ID":"ac1292b7-fafb-42ce-b43c-fe82b356ca0b","Type":"ContainerDied","Data":"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"}
Nov 02 08:27:11.482194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.481557    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8" event={"ID":"ac1292b7-fafb-42ce-b43c-fe82b356ca0b","Type":"ContainerDied","Data":"1372fc848f72b4a29217ceb43123ad578308604aed2d35d1d52655cb8d69d161"}
Nov 02 08:27:11.482194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.481590    4515 scope.go:122] "RemoveContainer" containerID="8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"
Nov 02 08:27:11.484576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.484552    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"
Nov 02 08:27:11.484576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.484560    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" event={"ID":"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a","Type":"ContainerDied","Data":"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"}
Nov 02 08:27:11.484760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.484565    4515 generic.go:356] "Generic (PLEG): container finished" podID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerID="97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f" exitCode=0
Nov 02 08:27:11.484760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.484681    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856" event={"ID":"02e1e52e-7379-4863-b1e3-3dfa3d1fc39a","Type":"ContainerDied","Data":"5619763e394548dcf8268b63cdf80fd71aff82786a33f5bf0ec5c20384224ffd"}
Nov 02 08:27:11.495692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.495574    4515 scope.go:122] "RemoveContainer" containerID="8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"
Nov 02 08:27:11.495989 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:27:11.495952    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8\": container with ID starting with 8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8 not found: ID does not exist" containerID="8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"
Nov 02 08:27:11.496125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.495994    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8"} err="failed to get container status \"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8\": rpc error: code = NotFound desc = could not find container \"8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8\": container with ID starting with 8788717e487a2acd2bea74df51cba71237ff7b1582f1d207a9001f0dd942f2c8 not found: ID does not exist"
Nov 02 08:27:11.496125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.496014    4515 scope.go:122] "RemoveContainer" containerID="97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"
Nov 02 08:27:11.507793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.507769    4515 scope.go:122] "RemoveContainer" containerID="97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"
Nov 02 08:27:11.508111 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:27:11.508085    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f\": container with ID starting with 97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f not found: ID does not exist" containerID="97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"
Nov 02 08:27:11.508206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.508120    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f"} err="failed to get container status \"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f\": rpc error: code = NotFound desc = could not find container \"97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f\": container with ID starting with 97fa3fde573849209bf911e309a07af0f121f5d0712f36e6b7f58f852a191a2f not found: ID does not exist"
Nov 02 08:27:11.542224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.542156    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:27:11.546749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.546721    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b72fa3-7895dcd6d7-pb4v8"]
Nov 02 08:27:11.581105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.581061    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:27:11.587096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:11.587071    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b72fa3-b745c9fc5-fn856"]
Nov 02 08:27:13.375261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:13.375221    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" path="/var/lib/kubelet/pods/02e1e52e-7379-4863-b1e3-3dfa3d1fc39a/volumes"
Nov 02 08:27:13.375803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:13.375785    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" path="/var/lib/kubelet/pods/ac1292b7-fafb-42ce-b43c-fe82b356ca0b/volumes"
Nov 02 08:27:17.723973 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:27:17.723930    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:27:18.289795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:18.289755    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:27:27.422291 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.422253    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:27:27.422778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.422555    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" podUID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" containerName="maas-api" containerID="cri-o://a3c6b3aaba524637365e3b60e0201e475184027280452564c6046fd6f01ae168" gracePeriod=30
Nov 02 08:27:27.605909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.605880    4515 generic.go:356] "Generic (PLEG): container finished" podID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" containerID="a3c6b3aaba524637365e3b60e0201e475184027280452564c6046fd6f01ae168" exitCode=0
Nov 02 08:27:27.606076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.605959    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" event={"ID":"ae26a955-a6d1-4db2-8772-fdb53c2d6d20","Type":"ContainerDied","Data":"a3c6b3aaba524637365e3b60e0201e475184027280452564c6046fd6f01ae168"}
Nov 02 08:27:27.722012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.721983    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:27:27.856265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.856220    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") pod \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") "
Nov 02 08:27:27.856567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.856332    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\" (UniqueName: \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\") pod \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") "
Nov 02 08:27:27.856567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.856500    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\") pod \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") "
Nov 02 08:27:27.856682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.856574    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\") pod \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\" (UID: \"ae26a955-a6d1-4db2-8772-fdb53c2d6d20\") "
Nov 02 08:27:27.856966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.856941    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca" pod "ae26a955-a6d1-4db2-8772-fdb53c2d6d20" (UID: "ae26a955-a6d1-4db2-8772-fdb53c2d6d20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:27.857369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.857347    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:27.858843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.858815    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs" pod "ae26a955-a6d1-4db2-8772-fdb53c2d6d20" (UID: "ae26a955-a6d1-4db2-8772-fdb53c2d6d20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:27:27.859057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.859029    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls" pod "ae26a955-a6d1-4db2-8772-fdb53c2d6d20" (UID: "ae26a955-a6d1-4db2-8772-fdb53c2d6d20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:27:27.859160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.859081    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l" pod "ae26a955-a6d1-4db2-8772-fdb53c2d6d20" (UID: "ae26a955-a6d1-4db2-8772-fdb53c2d6d20"). InnerVolumeSpecName "kube-api-access-r978l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:27.960389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.959589    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r978l\" (UniqueName: \"kubernetes.io/projected/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-kube-api-access-r978l\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:27.960389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.959626    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:27.960389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:27.959641    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae26a955-a6d1-4db2-8772-fdb53c2d6d20-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:28.614289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:28.614256    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k" event={"ID":"ae26a955-a6d1-4db2-8772-fdb53c2d6d20","Type":"ContainerDied","Data":"c4501f0273abf6c471019e3448a840e996872858f3230c79115ffc6e80b42667"}
Nov 02 08:27:28.614743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:28.614292    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"
Nov 02 08:27:28.614743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:28.614347    4515 scope.go:122] "RemoveContainer" containerID="a3c6b3aaba524637365e3b60e0201e475184027280452564c6046fd6f01ae168"
Nov 02 08:27:28.667570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:28.667537    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:27:28.680120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:28.680088    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b72fa3-6d86c8f8b4-k8v6k"]
Nov 02 08:27:29.375227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:29.375184    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" path="/var/lib/kubelet/pods/ae26a955-a6d1-4db2-8772-fdb53c2d6d20/volumes"
Nov 02 08:27:32.552800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:32.552762    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:27:32.553356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:32.553013    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy" containerID="cri-o://0fb9881d238265e706acab165d408b7a138637fdc66840e3e3f8665c121cb671" gracePeriod=30
Nov 02 08:27:34.056131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.056095    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:27:34.057026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057001    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057028    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057046    4515 cpu_manager.go:414] "containerMap: removing container" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057054    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057081    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057096    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057112    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" containerName="maas-api"
Nov 02 08:27:34.057195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057118    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" containerName="maas-api"
Nov 02 08:27:34.057593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057347    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1252f400-2f23-47b6-97c2-5e564b1725ff" containerName="istio-proxy"
Nov 02 08:27:34.057593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057378    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e1e52e-7379-4863-b1e3-3dfa3d1fc39a" containerName="payload-pre-processing"
Nov 02 08:27:34.057593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057390    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1292b7-fafb-42ce-b43c-fe82b356ca0b" containerName="payload-processing"
Nov 02 08:27:34.057593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.057400    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae26a955-a6d1-4db2-8772-fdb53c2d6d20" containerName="maas-api"
Nov 02 08:27:34.062780 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.062760    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.078201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.078175    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4b85d50c-openshift-default-dockercfg-svrnp\""
Nov 02 08:27:34.098293 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.098263    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:27:34.150388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150352    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150406    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150477    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150537    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4xk\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150693 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150633    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150693 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150663    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150719    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150752    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.150834 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.150781    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.255646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.255605    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.262982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.262897    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.270344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.266874    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274496    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274561    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274634    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4xk\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274701    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274730    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274791    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274821    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.274864    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.275964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.275703    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.276026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.275965    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.279334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.276541    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.279334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.276778    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.292333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.279972    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.324425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.308102    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4xk\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.324425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.312672    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\") pod \"e2e-mt-4b85d50c-openshift-default-558d957877-z2knj\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") " pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.377751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.377717    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:34.711516 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:27:34.711486    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1485c7f4_d98c_4219_af21_62bbe779f6a9.slice/crio-8a8945a9378baa10c50ad58b7973544f3d729df7de2a87db07e29f0a9cd32473 WatchSource:0}: Error finding container 8a8945a9378baa10c50ad58b7973544f3d729df7de2a87db07e29f0a9cd32473: Status 404 returned error can't find the container with id 8a8945a9378baa10c50ad58b7973544f3d729df7de2a87db07e29f0a9cd32473
Nov 02 08:27:34.713690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.713662    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:27:34.714900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.714866    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:27:34.714989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.714934    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:27:34.714989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:34.714964    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:27:35.665318 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:35.665268    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerStarted","Data":"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"}
Nov 02 08:27:35.665824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:35.665344    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerStarted","Data":"8a8945a9378baa10c50ad58b7973544f3d729df7de2a87db07e29f0a9cd32473"}
Nov 02 08:27:35.723347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:35.723281    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podStartSLOduration=1.72326757 podStartE2EDuration="1.72326757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:35.722644826 +0000 UTC m=+3800.485081729" watchObservedRunningTime="2026-11-02 08:27:35.72326757 +0000 UTC m=+3800.485704476"
Nov 02 08:27:36.378651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:36.378616    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:36.384897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:36.384868    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:36.672993 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:36.672900    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:36.673891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:36.673862    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:27:37.683101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.683071    4515 generic.go:356] "Generic (PLEG): container finished" podID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerID="0fb9881d238265e706acab165d408b7a138637fdc66840e3e3f8665c121cb671" exitCode=0
Nov 02 08:27:37.683491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.683159    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerDied","Data":"0fb9881d238265e706acab165d408b7a138637fdc66840e3e3f8665c121cb671"}
Nov 02 08:27:37.683491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.683236    4515 scope.go:122] "RemoveContainer" containerID="09c99179b623c4dc33e3524e41038823d6df7521537d0afd09a945ef1f21a12b"
Nov 02 08:27:37.707445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.707416    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:27:37.754467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.754433    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:27:37.865330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:37.862754    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:27:38.029820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029788    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029837    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029866    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029901    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029946    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.029991    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030107    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030241    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.030376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030251    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.030376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030286    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030333    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\") pod \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\" (UID: \"024ae50a-2682-4d40-9d8f-fbf460cdf97a\") "
Nov 02 08:27:38.030608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030414    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.030608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030446    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:38.030608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030519    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.031029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030971    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.031029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.030999    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.031029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.031014    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.031029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.031028    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.031357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.031041    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.032421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.032397    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.032421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.032409    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:27:38.032597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.032575    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:38.032706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.032691    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2" pod "024ae50a-2682-4d40-9d8f-fbf460cdf97a" (UID: "024ae50a-2682-4d40-9d8f-fbf460cdf97a"). InnerVolumeSpecName "kube-api-access-tpbt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:38.140522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.140293    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.140522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.140366    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpbt2\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-kube-api-access-tpbt2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.140522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.140378    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.140522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.140388    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ae50a-2682-4d40-9d8f-fbf460cdf97a-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.690957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.690921    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"
Nov 02 08:27:38.691442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.690955    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn" event={"ID":"024ae50a-2682-4d40-9d8f-fbf460cdf97a","Type":"ContainerDied","Data":"20db5d238876fc6aacf8b9c7cc699b0803a75a6e01bf5d671da9d51b4941ce5c"}
Nov 02 08:27:38.691442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.691016    4515 scope.go:122] "RemoveContainer" containerID="0fb9881d238265e706acab165d408b7a138637fdc66840e3e3f8665c121cb671"
Nov 02 08:27:38.772561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.772521    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:27:38.788353 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:38.787653    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-nkcxn"]
Nov 02 08:27:39.375055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:39.375020    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" path="/var/lib/kubelet/pods/024ae50a-2682-4d40-9d8f-fbf460cdf97a/volumes"
Nov 02 08:27:40.201459 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.201425    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:27:40.203546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203510    4515 cpu_manager.go:414] "containerMap: removing container" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.203546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203545    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.203756 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203572    4515 cpu_manager.go:414] "containerMap: removing container" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.203756 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203580    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.203956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203940    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.204017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.203961    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="024ae50a-2682-4d40-9d8f-fbf460cdf97a" containerName="istio-proxy"
Nov 02 08:27:40.216219 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.216192    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.217680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.217652    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:27:40.222872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.222847    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.234304 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.234277    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:27:40.249580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.249550    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4b85d50c\""
Nov 02 08:27:40.249580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.249554    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4b85d50c-dockercfg-nrbwr\""
Nov 02 08:27:40.251123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.251088    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:27:40.271221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.271186    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnht\" (UniqueName: \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.271435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.271246    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.271435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.271410    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d25\" (UniqueName: \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.271583 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.271442    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.383397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.383357    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnht\" (UniqueName: \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.383719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.383694    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.383951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.383934    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d25\" (UniqueName: \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.384068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.384053    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.396330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.389345    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.398419 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.398388    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.461385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.447052    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnht\" (UniqueName: \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\") pod \"payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") " pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.461385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.447052    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d25\" (UniqueName: \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\") pod \"payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.537524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.537480    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:40.553847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.553824    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:40.859632 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:27:40.859602    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ce8faa_86cd_4d3a_bb6d_794cbb5d47c6.slice/crio-b8b063740460670578554b6440d1636dbdca072151eee60ca371111fc201548b WatchSource:0}: Error finding container b8b063740460670578554b6440d1636dbdca072151eee60ca371111fc201548b: Status 404 returned error can't find the container with id b8b063740460670578554b6440d1636dbdca072151eee60ca371111fc201548b
Nov 02 08:27:40.860841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.860811    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:27:40.893476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:40.893446    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:27:40.896931 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:27:40.896906    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fba58e_785e_4418_a949_d99701bff54f.slice/crio-bede0c35f773011219f95af339711e7e887b49712d94c29961e2a4fc6122b8f5 WatchSource:0}: Error finding container bede0c35f773011219f95af339711e7e887b49712d94c29961e2a4fc6122b8f5: Status 404 returned error can't find the container with id bede0c35f773011219f95af339711e7e887b49712d94c29961e2a4fc6122b8f5
Nov 02 08:27:41.704344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.704289    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:27:41.704886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.704526    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" containerID="cri-o://546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c" gracePeriod=30
Nov 02 08:27:41.746611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.746569    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" event={"ID":"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6","Type":"ContainerStarted","Data":"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"}
Nov 02 08:27:41.746834 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.746635    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" event={"ID":"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6","Type":"ContainerStarted","Data":"b8b063740460670578554b6440d1636dbdca072151eee60ca371111fc201548b"}
Nov 02 08:27:41.746834 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.746668    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:41.748371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.748338    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" event={"ID":"98fba58e-785e-4418-a949-d99701bff54f","Type":"ContainerStarted","Data":"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"}
Nov 02 08:27:41.748516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.748391    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" event={"ID":"98fba58e-785e-4418-a949-d99701bff54f","Type":"ContainerStarted","Data":"bede0c35f773011219f95af339711e7e887b49712d94c29961e2a4fc6122b8f5"}
Nov 02 08:27:41.748516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.748416    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:41.813332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.811595    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" podStartSLOduration=1.811579773 podStartE2EDuration="1.811579773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:41.810790627 +0000 UTC m=+3806.573227532" watchObservedRunningTime="2026-11-02 08:27:41.811579773 +0000 UTC m=+3806.574016679"
Nov 02 08:27:41.874333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:41.873933    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" podStartSLOduration=1.873911525 podStartE2EDuration="1.873911525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:27:41.866712052 +0000 UTC m=+3806.629148959" watchObservedRunningTime="2026-11-02 08:27:41.873911525 +0000 UTC m=+3806.636348433"
Nov 02 08:27:42.708496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:42.708440    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing" containerID="cri-o://2ed6438774a9b5024cc38f3610de31a0019c8f704f032cefab9205c55f5dc88f" gracePeriod=25
Nov 02 08:27:42.755097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:42.755054    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing" containerID="cri-o://d129c2dd6bc7400bcc1f403f17442e1d388dc815afb2fd264134f8508e15f400" gracePeriod=25
Nov 02 08:27:44.959144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:44.959101    4515 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:27:44.959633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:44.959169    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:27:46.976631 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:46.976565    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:27:47.097430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097389    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097473    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097508    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097566    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097660    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097724    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.097802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097726    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.097802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097793    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.098040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.097833    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.098040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098022    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\") pod \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\" (UID: \"702facf3-39ec-4bb2-ba2c-c898a6f9dc15\") "
Nov 02 08:27:47.098454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098235    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.098620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098598    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:47.098779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098712    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.098923 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098774    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.099046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.098825    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.099162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.099147    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.100432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.100294    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:47.100432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.100408    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "kube-api-access-mjjp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:47.100717 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.100684    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:27:47.100825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.100786    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy" pod "702facf3-39ec-4bb2-ba2c-c898a6f9dc15" (UID: "702facf3-39ec-4bb2-ba2c-c898a6f9dc15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.200784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200748    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.200784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200780    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.200784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200789    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjjp7\" (UniqueName: \"kubernetes.io/projected/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-kube-api-access-mjjp7\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.201057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200798    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.201057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200807    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.201057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200815    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.201057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.200823    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702facf3-39ec-4bb2-ba2c-c898a6f9dc15-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.725262 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:27:47.725177    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:27:47.792629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.792593    4515 generic.go:356] "Generic (PLEG): container finished" podID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerID="546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c" exitCode=0
Nov 02 08:27:47.792804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.792643    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"
Nov 02 08:27:47.792804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.792673    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" event={"ID":"702facf3-39ec-4bb2-ba2c-c898a6f9dc15","Type":"ContainerDied","Data":"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"}
Nov 02 08:27:47.792804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.792732    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47" event={"ID":"702facf3-39ec-4bb2-ba2c-c898a6f9dc15","Type":"ContainerDied","Data":"b6f8e8f54babe42c2874f770a3ebbc7a3de6a76a3f5a5ec5cd384461464052ec"}
Nov 02 08:27:47.792804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.792756    4515 scope.go:122] "RemoveContainer" containerID="546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"
Nov 02 08:27:47.804064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.804040    4515 scope.go:122] "RemoveContainer" containerID="546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"
Nov 02 08:27:47.804405 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:27:47.804384    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c\": container with ID starting with 546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c not found: ID does not exist" containerID="546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"
Nov 02 08:27:47.804456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.804416    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c"} err="failed to get container status \"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c\": rpc error: code = NotFound desc = could not find container \"546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c\": container with ID starting with 546f7bd2830bc0957437a77dd6f949b552ba8c420a5e43c205f63fa27876d42c not found: ID does not exist"
Nov 02 08:27:47.865931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.865898    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:27:47.869517 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:47.869488    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-pdj47"]
Nov 02 08:27:49.374374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:49.374305    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" path="/var/lib/kubelet/pods/702facf3-39ec-4bb2-ba2c-c898a6f9dc15/volumes"
Nov 02 08:27:49.378703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:49.378674    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:49.378868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:49.378674    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:52.759802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:52.759772    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:27:52.760350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:52.759848    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:27:59.379416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:59.379331    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:59.379905 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:27:59.379328    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:28:02.898090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:02.898054    4515 generic.go:356] "Generic (PLEG): container finished" podID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerID="2ed6438774a9b5024cc38f3610de31a0019c8f704f032cefab9205c55f5dc88f" exitCode=0
Nov 02 08:28:02.898559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:02.898133    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" event={"ID":"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec","Type":"ContainerDied","Data":"2ed6438774a9b5024cc38f3610de31a0019c8f704f032cefab9205c55f5dc88f"}
Nov 02 08:28:02.900014 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:02.899988    4515 generic.go:356] "Generic (PLEG): container finished" podID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerID="d129c2dd6bc7400bcc1f403f17442e1d388dc815afb2fd264134f8508e15f400" exitCode=0
Nov 02 08:28:02.900157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:02.900073    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" event={"ID":"f3408354-361c-4d06-ad38-b3271fbc7d25","Type":"ContainerDied","Data":"d129c2dd6bc7400bcc1f403f17442e1d388dc815afb2fd264134f8508e15f400"}
Nov 02 08:28:03.039284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.039256    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:28:03.087234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.087199    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:28:03.141755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.141720    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\" (UniqueName: \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\") pod \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") "
Nov 02 08:28:03.141933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.141820    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\" (UniqueName: \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\") pod \"f3408354-361c-4d06-ad38-b3271fbc7d25\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") "
Nov 02 08:28:03.142125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142066    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\") pod \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\" (UID: \"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec\") "
Nov 02 08:28:03.142231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142205    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\") pod \"f3408354-361c-4d06-ad38-b3271fbc7d25\" (UID: \"f3408354-361c-4d06-ad38-b3271fbc7d25\") "
Nov 02 08:28:03.142611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142586    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume" pod "6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" (UID: "6a6a1263-e25f-430c-a3eb-9817ff0ca3ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:28:03.142755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142729    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume" pod "f3408354-361c-4d06-ad38-b3271fbc7d25" (UID: "f3408354-361c-4d06-ad38-b3271fbc7d25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:28:03.142901 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142883    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3408354-361c-4d06-ad38-b3271fbc7d25-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:28:03.142970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.142907    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:28:03.144612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.144587    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4" pod "6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" (UID: "6a6a1263-e25f-430c-a3eb-9817ff0ca3ec"). InnerVolumeSpecName "kube-api-access-hf4b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:28:03.145222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.145204    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf" pod "f3408354-361c-4d06-ad38-b3271fbc7d25" (UID: "f3408354-361c-4d06-ad38-b3271fbc7d25"). InnerVolumeSpecName "kube-api-access-bs5nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:28:03.244527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.244428    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4b4\" (UniqueName: \"kubernetes.io/projected/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec-kube-api-access-hf4b4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:28:03.244527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.244465    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5nf\" (UniqueName: \"kubernetes.io/projected/f3408354-361c-4d06-ad38-b3271fbc7d25-kube-api-access-bs5nf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:28:03.908395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.908364    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"
Nov 02 08:28:03.908395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.908379    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2" event={"ID":"6a6a1263-e25f-430c-a3eb-9817ff0ca3ec","Type":"ContainerDied","Data":"fb22336fc08df52606ad05b031e1ce7029e65c36e456d3860d353a33a976d724"}
Nov 02 08:28:03.908966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.908446    4515 scope.go:122] "RemoveContainer" containerID="2ed6438774a9b5024cc38f3610de31a0019c8f704f032cefab9205c55f5dc88f"
Nov 02 08:28:03.910009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.909988    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"
Nov 02 08:28:03.910129 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.910017    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f" event={"ID":"f3408354-361c-4d06-ad38-b3271fbc7d25","Type":"ContainerDied","Data":"e969b9aa173fbb812feb530c267f99d34928e7e9a1789e1634a7dd3c414b4122"}
Nov 02 08:28:03.919220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.919204    4515 scope.go:122] "RemoveContainer" containerID="d129c2dd6bc7400bcc1f403f17442e1d388dc815afb2fd264134f8508e15f400"
Nov 02 08:28:03.962836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.962805    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:28:03.975527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:03.975499    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-eb6010-59f8cf7dd7-pj4g2"]
Nov 02 08:28:04.011674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:04.011642    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:28:04.018790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:04.018750    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-eb6010-576fcf969f-jm79f"]
Nov 02 08:28:05.375275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:05.375238    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" path="/var/lib/kubelet/pods/6a6a1263-e25f-430c-a3eb-9817ff0ca3ec/volumes"
Nov 02 08:28:05.375712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:05.375663    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" path="/var/lib/kubelet/pods/f3408354-361c-4d06-ad38-b3271fbc7d25/volumes"
Nov 02 08:28:09.192938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:09.192899    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:28:17.726501 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:28:17.726463    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:28:45.488043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488005    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:28:45.488749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488730    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing"
Nov 02 08:28:45.488749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488749    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing"
Nov 02 08:28:45.488911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488765    4515 cpu_manager.go:414] "containerMap: removing container" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy"
Nov 02 08:28:45.488911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488770    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy"
Nov 02 08:28:45.488911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488784    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing"
Nov 02 08:28:45.488911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488789    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing"
Nov 02 08:28:45.489147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488936    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3408354-361c-4d06-ad38-b3271fbc7d25" containerName="payload-pre-processing"
Nov 02 08:28:45.489147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488946    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="702facf3-39ec-4bb2-ba2c-c898a6f9dc15" containerName="istio-proxy"
Nov 02 08:28:45.489147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.488955    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6a1263-e25f-430c-a3eb-9817ff0ca3ec" containerName="payload-processing"
Nov 02 08:28:45.493428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.493401    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.509658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.509633    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"kube-root-ca.crt\""
Nov 02 08:28:45.521702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.521669    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"default-dockercfg-9b42f\""
Nov 02 08:28:45.522027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.522005    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"sub-test-model-7f19e8-kserve-self-signed-certs\""
Nov 02 08:28:45.522184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.522105    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7f19e8\"/\"openshift-service-ca.crt\""
Nov 02 08:28:45.526689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.526665    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:28:45.585990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.585951    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.586186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.586074    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.586186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.586147    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.586186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.586166    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.586399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.586298    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.586456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.586403    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw22m\" (UniqueName: \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.687531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.687491    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.687719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.687598    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw22m\" (UniqueName: \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.687806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.687780    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.687962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.687893    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.688042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.687954    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.688112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.688056    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.688112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.688073    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.688112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.688089    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.688361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.688299    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.690273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.690245    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.690930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.690908    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.742415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.728279    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw22m\" (UniqueName: \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\") pod \"sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") " pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:45.813755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:45.813711    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:46.123792 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:28:46.123732    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod117b6b36_a1e8_4369_9ae9_e4d2f7299419.slice/crio-d6510e3d33dd62dda2cabf0fe44acf469d42cb5673c1cfea2a5ceae3c9a05ebb WatchSource:0}: Error finding container d6510e3d33dd62dda2cabf0fe44acf469d42cb5673c1cfea2a5ceae3c9a05ebb: Status 404 returned error can't find the container with id d6510e3d33dd62dda2cabf0fe44acf469d42cb5673c1cfea2a5ceae3c9a05ebb
Nov 02 08:28:46.124772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:46.124745    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:28:46.273349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:46.273287    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:28:46.294811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:46.294777    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" event={"ID":"117b6b36-a1e8-4369-9ae9-e4d2f7299419","Type":"ContainerStarted","Data":"d6510e3d33dd62dda2cabf0fe44acf469d42cb5673c1cfea2a5ceae3c9a05ebb"}
Nov 02 08:28:47.304117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:47.304079    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" event={"ID":"117b6b36-a1e8-4369-9ae9-e4d2f7299419","Type":"ContainerStarted","Data":"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"}
Nov 02 08:28:47.360462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:47.360411    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" podStartSLOduration=2.360396852 podStartE2EDuration="2.360396852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:47.358673481 +0000 UTC m=+3872.121110388" watchObservedRunningTime="2026-11-02 08:28:47.360396852 +0000 UTC m=+3872.122833757"
Nov 02 08:28:47.754129 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:28:47.754082    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:28:47.814498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:47.814466    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:55.817408 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:55.817371    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:55.839358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:55.839291    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:56.406524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:56.406490    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:28:57.403795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.403689    4515 generic.go:356] "Generic (PLEG): container finished" podID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerID="25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66" exitCode=0
Nov 02 08:28:57.403795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.403763    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerDied","Data":"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"}
Nov 02 08:28:57.404432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.404237    4515 scope.go:122] "RemoveContainer" containerID="25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"
Nov 02 08:28:57.406081 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.406051    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:28:57.406168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.406121    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:28:57.406168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:57.406154    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:28:58.415076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:58.415041    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerStarted","Data":"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"}
Nov 02 08:28:59.378461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:59.378429    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:28:59.378624 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:59.378604    4515 patch_prober.go:29] interesting pod/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 02 08:28:59.378686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:28:59.378659    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 02 08:29:00.379910 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:00.379871    4515 patch_prober.go:29] interesting pod/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 02 08:29:00.380358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:00.379935    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 02 08:29:01.378447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:01.378413    4515 patch_prober.go:29] interesting pod/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 02 08:29:01.378633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:01.378480    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 02 08:29:02.378320 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:02.378270    4515 patch_prober.go:29] interesting pod/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 02 08:29:02.378734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:02.378364    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 02 08:29:03.057215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:03.056392    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:29:03.057215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:03.056655    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" containerID="cri-o://66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc" gracePeriod=30
Nov 02 08:29:03.378627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:03.378546    4515 patch_prober.go:29] interesting pod/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 02 08:29:03.378627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:03.378599    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 02 08:29:04.383538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:04.383505    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:29:04.384025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:04.383987    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:29:04.384928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:04.384906    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:29:05.643528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:05.643489    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:29:05.644028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:05.643544    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:29:08.324161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.324094    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:29:08.383008 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.382964    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383026    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383072    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383135    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383250    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383407    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383440    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383488    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383527    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:29:08.383637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383441    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:08.383637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383540    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:29:08.383847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383653    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\") pod \"72a06d99-ea63-49a8-ab4d-e26185d54896\" (UID: \"72a06d99-ea63-49a8-ab4d-e26185d54896\") "
Nov 02 08:29:08.383847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383663    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:29:08.383975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.383952    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:29:08.384322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.384289    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.384440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.384327    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.384440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.384341    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.384440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.384353    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.384440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.384366    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a06d99-ea63-49a8-ab4d-e26185d54896-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.385869 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.385838    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "kube-api-access-46qvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:08.385999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.385917    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:29:08.386071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.386029    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:29:08.386136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.386075    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token" pod "72a06d99-ea63-49a8-ab4d-e26185d54896" (UID: "72a06d99-ea63-49a8-ab4d-e26185d54896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:08.485589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.485554    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.485589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.485585    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qvv\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-kube-api-access-46qvv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.485589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.485596    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.485829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.485605    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a06d99-ea63-49a8-ab4d-e26185d54896-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:08.498720 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.498685    4515 generic.go:356] "Generic (PLEG): container finished" podID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerID="66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc" exitCode=0
Nov 02 08:29:08.498881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.498744    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"
Nov 02 08:29:08.498881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.498758    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerDied","Data":"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"}
Nov 02 08:29:08.498881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.498803    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z" event={"ID":"72a06d99-ea63-49a8-ab4d-e26185d54896","Type":"ContainerDied","Data":"a760614990b3e09d45d7302771920b8bd9fcd621833d16014fced6ce76a351fa"}
Nov 02 08:29:08.498881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.498823    4515 scope.go:122] "RemoveContainer" containerID="66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"
Nov 02 08:29:08.510448 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.510421    4515 scope.go:122] "RemoveContainer" containerID="0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"
Nov 02 08:29:08.520711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.520681    4515 scope.go:122] "RemoveContainer" containerID="66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"
Nov 02 08:29:08.521009 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:29:08.520990    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc\": container with ID starting with 66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc not found: ID does not exist" containerID="66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"
Nov 02 08:29:08.521078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.521019    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc"} err="failed to get container status \"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc\": rpc error: code = NotFound desc = could not find container \"66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc\": container with ID starting with 66df2eae86db187432f8f7b869c68835442d2c9c3048e3cdf609b22e108973bc not found: ID does not exist"
Nov 02 08:29:08.521078 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.521037    4515 scope.go:122] "RemoveContainer" containerID="0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"
Nov 02 08:29:08.521274 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:29:08.521256    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d\": container with ID starting with 0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d not found: ID does not exist" containerID="0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"
Nov 02 08:29:08.521339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.521279    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d"} err="failed to get container status \"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d\": rpc error: code = NotFound desc = could not find container \"0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d\": container with ID starting with 0f663464343b3872a630d0dd5bf8ad4b8004381bc3712a6f817684243519028d not found: ID does not exist"
Nov 02 08:29:08.555869 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.555836    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:29:08.560381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:08.560357    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-736f7c-openshift-default-56685bb6d5-gwn2z"]
Nov 02 08:29:09.376889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:09.376846    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" path="/var/lib/kubelet/pods/72a06d99-ea63-49a8-ab4d-e26185d54896/volumes"
Nov 02 08:29:10.605916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:10.605881    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:29:10.641233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:10.641195    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:29:15.606384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:15.606348    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing" containerID="cri-o://e65a75946d6fa579469a2fb2fd0930ae012281930df25bd39397201da5bff096" gracePeriod=25
Nov 02 08:29:15.642621 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:15.642550    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing" containerID="cri-o://5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9" gracePeriod=25
Nov 02 08:29:17.726243 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:29:17.726198    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:29:19.886013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:19.885973    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:19.886487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:19.886042    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:29.885584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:29.885499    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:29.885983 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:29.885522    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:35.734157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:35.734117    4515 generic.go:356] "Generic (PLEG): container finished" podID="b9670062-f9c8-4486-87db-207a93997f16" containerID="e65a75946d6fa579469a2fb2fd0930ae012281930df25bd39397201da5bff096" exitCode=0
Nov 02 08:29:35.734550 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:35.734189    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" event={"ID":"b9670062-f9c8-4486-87db-207a93997f16","Type":"ContainerDied","Data":"e65a75946d6fa579469a2fb2fd0930ae012281930df25bd39397201da5bff096"}
Nov 02 08:29:35.967013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:35.966991    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:29:35.970471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:35.970450    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:29:36.078682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.078652    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\" (UniqueName: \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\") pod \"b9670062-f9c8-4486-87db-207a93997f16\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") "
Nov 02 08:29:36.078877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.078698    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\") pod \"d607758e-3dfe-4698-95d2-c79123967559\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") "
Nov 02 08:29:36.078877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.078743    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\" (UniqueName: \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\") pod \"d607758e-3dfe-4698-95d2-c79123967559\" (UID: \"d607758e-3dfe-4698-95d2-c79123967559\") "
Nov 02 08:29:36.079006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.078912    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\") pod \"b9670062-f9c8-4486-87db-207a93997f16\" (UID: \"b9670062-f9c8-4486-87db-207a93997f16\") "
Nov 02 08:29:36.079329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.079273    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume" pod "d607758e-3dfe-4698-95d2-c79123967559" (UID: "d607758e-3dfe-4698-95d2-c79123967559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:36.079475 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.079357    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume" pod "b9670062-f9c8-4486-87db-207a93997f16" (UID: "b9670062-f9c8-4486-87db-207a93997f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:36.079553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.079494    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670062-f9c8-4486-87db-207a93997f16-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:36.079553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.079512    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d607758e-3dfe-4698-95d2-c79123967559-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:36.081560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.081536    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx" pod "b9670062-f9c8-4486-87db-207a93997f16" (UID: "b9670062-f9c8-4486-87db-207a93997f16"). InnerVolumeSpecName "kube-api-access-rdtcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:36.081689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.081565    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4" pod "d607758e-3dfe-4698-95d2-c79123967559" (UID: "d607758e-3dfe-4698-95d2-c79123967559"). InnerVolumeSpecName "kube-api-access-rnck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:36.180645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.180568    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdtcx\" (UniqueName: \"kubernetes.io/projected/b9670062-f9c8-4486-87db-207a93997f16-kube-api-access-rdtcx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:36.180645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.180596    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnck4\" (UniqueName: \"kubernetes.io/projected/d607758e-3dfe-4698-95d2-c79123967559-kube-api-access-rnck4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:29:36.741733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.741690    4515 generic.go:356] "Generic (PLEG): container finished" podID="d607758e-3dfe-4698-95d2-c79123967559" containerID="5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9" exitCode=0
Nov 02 08:29:36.741733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.741738    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"
Nov 02 08:29:36.742257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.741770    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" event={"ID":"d607758e-3dfe-4698-95d2-c79123967559","Type":"ContainerDied","Data":"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"}
Nov 02 08:29:36.742257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.741830    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss" event={"ID":"d607758e-3dfe-4698-95d2-c79123967559","Type":"ContainerDied","Data":"69d24a2bc89423c89f0ff21611e17de6c3a490ced943dda7e6b791c46009c49a"}
Nov 02 08:29:36.742257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.741860    4515 scope.go:122] "RemoveContainer" containerID="5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"
Nov 02 08:29:36.743411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.743388    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"
Nov 02 08:29:36.743528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.743427    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98" event={"ID":"b9670062-f9c8-4486-87db-207a93997f16","Type":"ContainerDied","Data":"8b22421928a3045ac2263acc7bc7f5d097a8101cfc1704906312bbac016f8502"}
Nov 02 08:29:36.753664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.753476    4515 scope.go:122] "RemoveContainer" containerID="5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"
Nov 02 08:29:36.753773 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:29:36.753751    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9\": container with ID starting with 5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9 not found: ID does not exist" containerID="5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"
Nov 02 08:29:36.753820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.753782    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9"} err="failed to get container status \"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9\": rpc error: code = NotFound desc = could not find container \"5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9\": container with ID starting with 5794159123f348f76511c1f5c023387f1a96191b5a91172ef1e01360364575d9 not found: ID does not exist"
Nov 02 08:29:36.753820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.753800    4515 scope.go:122] "RemoveContainer" containerID="e65a75946d6fa579469a2fb2fd0930ae012281930df25bd39397201da5bff096"
Nov 02 08:29:36.800484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.800445    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:29:36.813134 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.813096    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2dde84-5d85974555-4pg98"]
Nov 02 08:29:36.842065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.842028    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:29:36.845371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:36.845332    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2dde84-6c4f95c467-pj9ss"]
Nov 02 08:29:37.374166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:37.374131    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9670062-f9c8-4486-87db-207a93997f16" path="/var/lib/kubelet/pods/b9670062-f9c8-4486-87db-207a93997f16/volumes"
Nov 02 08:29:37.374642 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:37.374627    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d607758e-3dfe-4698-95d2-c79123967559" path="/var/lib/kubelet/pods/d607758e-3dfe-4698-95d2-c79123967559/volumes"
Nov 02 08:29:42.637430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:29:42.637391    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:29:47.722991 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:29:47.722954    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:30:00.171458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.171406    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:00.172515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172486    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing"
Nov 02 08:30:00.172515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172511    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing"
Nov 02 08:30:00.172733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172533    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing"
Nov 02 08:30:00.172733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172538    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing"
Nov 02 08:30:00.172733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172557    4515 cpu_manager.go:414] "containerMap: removing container" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.172733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172564    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.172733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172734    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d607758e-3dfe-4698-95d2-c79123967559" containerName="payload-pre-processing"
Nov 02 08:30:00.172976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172752    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.172976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172764    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.172976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.172771    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9670062-f9c8-4486-87db-207a93997f16" containerName="payload-processing"
Nov 02 08:30:00.177274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.177250    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:00.177544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.177524    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.178346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.178230    4515 cpu_manager.go:414] "containerMap: removing container" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.178346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.178251    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a06d99-ea63-49a8-ab4d-e26185d54896" containerName="istio-proxy"
Nov 02 08:30:00.182576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.182549    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.209028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.208687    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:00.217779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.217587    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:00.222396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.222365    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:00.222604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.222574    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.222820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.222799    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.222926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.222908    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjg4\" (UniqueName: \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.223009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.222990    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jj2\" (UniqueName: \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.236056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.228369    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:00.236056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.228981    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.236056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.234739    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.250728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.250688    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:00.259153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.259117    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:00.260763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.260718    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.277490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.276245    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:30:00.277798 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.277700    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.286547 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.286519    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:00.293670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.293513    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.298356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.298291    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:00.305421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.305391    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:00.306176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.306155    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:00.312162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.312141    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:00.312294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.312248    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.319244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.317355    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:00.320984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.320962    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:00.325924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.325887    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.326063 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.325948    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:00.326148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326130    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qh6\" (UniqueName: \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326210    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326271    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326331    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjg4\" (UniqueName: \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326446    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jj2\" (UniqueName: \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326510    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj8c\" (UniqueName: \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.326925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.326547    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.327368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.327185    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.327368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.327203    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.351177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.351139    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjg4\" (UniqueName: \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.353959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.353932    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jj2\" (UniqueName: \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.428404 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428276    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj8c\" (UniqueName: \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.428404 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428333    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.428653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428436    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.428653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428483    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.428653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428548    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4md\" (UniqueName: \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.428653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428606    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplb6\" (UniqueName: \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.428836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428713    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qh6\" (UniqueName: \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.428890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.428837    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mxv\" (UniqueName: \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.429066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429044    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.429138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429042    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp28n\" (UniqueName: \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.429195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429177    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.429253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429210    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.429349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429254    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.429803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.429782    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.454146 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.454114    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj8c\" (UniqueName: \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.455651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.455629    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qh6\" (UniqueName: \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\") pod \"maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.510762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.510730    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:00.519474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.519449    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530644    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530710    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530743    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4md\" (UniqueName: \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530820    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplb6\" (UniqueName: \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530858    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mxv\" (UniqueName: \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530886    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp28n\" (UniqueName: \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.530996    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.531030    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.531490    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.531828    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.532115    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.532187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.532146    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.555215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.555160    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mxv\" (UniqueName: \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.555393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.555293    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp28n\" (UniqueName: \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\") pod \"maas-api-key-cleanup-29893470-nqn8c\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.555838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.555807    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplb6\" (UniqueName: \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.556044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.556024    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4md\" (UniqueName: \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.562847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.562822    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:00.581664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.581618    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:00.592517 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.591903    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:00.611095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.610879    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:00.625130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.624789    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:00.670513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.651044    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:00.842670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.841389    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:00.851423 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:00.851344    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78369e14_b18f_4ad8_a30b_32c1b99b49d7.slice/crio-828a22cd06d19fed4f144b4072bf062148f0d6328a617d297546e03b920e5c1a WatchSource:0}: Error finding container 828a22cd06d19fed4f144b4072bf062148f0d6328a617d297546e03b920e5c1a: Status 404 returned error can't find the container with id 828a22cd06d19fed4f144b4072bf062148f0d6328a617d297546e03b920e5c1a
Nov 02 08:30:00.933858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:00.933818    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerStarted","Data":"828a22cd06d19fed4f144b4072bf062148f0d6328a617d297546e03b920e5c1a"}
Nov 02 08:30:01.058905 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:01.058872    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f5a163_b184_4d6c_839c_9f239200f19a.slice/crio-f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79 WatchSource:0}: Error finding container f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79: Status 404 returned error can't find the container with id f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79
Nov 02 08:30:01.060641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.060618    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:01.084844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.084816    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:01.101987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.101954    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:01.140842 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:01.140650    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43443cca_aef0_479b_9fd2_c7545cdd676a.slice/crio-ec2975ea8593af29428a0f7599608dd0ab1f639e110b3c25a481e00a0e67bf48 WatchSource:0}: Error finding container ec2975ea8593af29428a0f7599608dd0ab1f639e110b3c25a481e00a0e67bf48: Status 404 returned error can't find the container with id ec2975ea8593af29428a0f7599608dd0ab1f639e110b3c25a481e00a0e67bf48
Nov 02 08:30:01.142239 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.142212    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:01.153590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.153562    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:01.163755 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:01.163720    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3596205_93a5_4e51_962b_9e49ccef4aa8.slice/crio-17356bd456e46abd6ef584c9d1927de1ac331770a2b887ceb15dcc3d8b38b778 WatchSource:0}: Error finding container 17356bd456e46abd6ef584c9d1927de1ac331770a2b887ceb15dcc3d8b38b778: Status 404 returned error can't find the container with id 17356bd456e46abd6ef584c9d1927de1ac331770a2b887ceb15dcc3d8b38b778
Nov 02 08:30:01.165485 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:01.165228    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7ea61f_ef56_4699_a8df_a84332efb81e.slice/crio-97ca66b038487aac517d84eb89279de0a7df0b6eeb7b661f1eaa1d29cce630ce WatchSource:0}: Error finding container 97ca66b038487aac517d84eb89279de0a7df0b6eeb7b661f1eaa1d29cce630ce: Status 404 returned error can't find the container with id 97ca66b038487aac517d84eb89279de0a7df0b6eeb7b661f1eaa1d29cce630ce
Nov 02 08:30:01.168232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.168210    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:01.181169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:01.181143    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:01.181542 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:30:01.181191    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b45d301_a087_40d2_8b4e_4f49130cb648.slice/crio-9b38adfc773e76e9a455c186070c755e6af2d3b3026c3bc443336de2f54d14f8 WatchSource:0}: Error finding container 9b38adfc773e76e9a455c186070c755e6af2d3b3026c3bc443336de2f54d14f8: Status 404 returned error can't find the container with id 9b38adfc773e76e9a455c186070c755e6af2d3b3026c3bc443336de2f54d14f8
Nov 02 08:30:02.316685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.291220    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerStarted","Data":"1cc90c977f74197dd11b9796a352f394e4e15c57e4d189ad4ed597a1b5267060"}
Nov 02 08:30:02.316685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.294585    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerStarted","Data":"6777ecf6ef264d5ae7e070f248a460feeaad5688844c327d4027c5e97f02b0f7"}
Nov 02 08:30:02.316685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.309708    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerStarted","Data":"2d68c23572684451a7b74f0631949ed793c16a53e93f70e57948a1a2edd6b402"}
Nov 02 08:30:02.426925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.423449    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerStarted","Data":"97ca66b038487aac517d84eb89279de0a7df0b6eeb7b661f1eaa1d29cce630ce"}
Nov 02 08:30:02.428301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.428067    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerStarted","Data":"ec2975ea8593af29428a0f7599608dd0ab1f639e110b3c25a481e00a0e67bf48"}
Nov 02 08:30:02.433398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.432972    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerStarted","Data":"17356bd456e46abd6ef584c9d1927de1ac331770a2b887ceb15dcc3d8b38b778"}
Nov 02 08:30:02.435612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.435537    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerStarted","Data":"9b38adfc773e76e9a455c186070c755e6af2d3b3026c3bc443336de2f54d14f8"}
Nov 02 08:30:02.450980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.450467    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerStarted","Data":"4b2591797df6378381562341fb32cd179591c2591e5d73c911d5edd0d3f4031a"}
Nov 02 08:30:02.450980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.450538    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerStarted","Data":"f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79"}
Nov 02 08:30:02.509301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.503331    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" podStartSLOduration=2.503293026 podStartE2EDuration="2.503293026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:02.393998515 +0000 UTC m=+3947.156435423" watchObservedRunningTime="2026-11-02 08:30:02.503293026 +0000 UTC m=+3947.265729939"
Nov 02 08:30:02.538052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.537991    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" podStartSLOduration=2.537964672 podStartE2EDuration="2.537964672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:02.50299726 +0000 UTC m=+3947.265434167" watchObservedRunningTime="2026-11-02 08:30:02.537964672 +0000 UTC m=+3947.300401579"
Nov 02 08:30:02.558859 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.558757    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" podStartSLOduration=2.5587404190000003 podStartE2EDuration="2.558740419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:02.548109224 +0000 UTC m=+3947.310546131" watchObservedRunningTime="2026-11-02 08:30:02.558740419 +0000 UTC m=+3947.321177325"
Nov 02 08:30:02.741211 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.741134    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" podStartSLOduration=2.741102533 podStartE2EDuration="2.741102533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:02.63978566 +0000 UTC m=+3947.402222568" watchObservedRunningTime="2026-11-02 08:30:02.741102533 +0000 UTC m=+3947.503539440"
Nov 02 08:30:02.829845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:02.829776    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" podStartSLOduration=2.829755677 podStartE2EDuration="2.829755677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:02.697391641 +0000 UTC m=+3947.459828548" watchObservedRunningTime="2026-11-02 08:30:02.829755677 +0000 UTC m=+3947.592192585"
Nov 02 08:30:03.526106 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.526053    4515 generic.go:356] "Generic (PLEG): container finished" podID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerID="2d68c23572684451a7b74f0631949ed793c16a53e93f70e57948a1a2edd6b402" exitCode=60
Nov 02 08:30:03.526813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.526780    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerDied","Data":"2d68c23572684451a7b74f0631949ed793c16a53e93f70e57948a1a2edd6b402"}
Nov 02 08:30:03.527508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.527487    4515 scope.go:122] "RemoveContainer" containerID="2d68c23572684451a7b74f0631949ed793c16a53e93f70e57948a1a2edd6b402"
Nov 02 08:30:03.559349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.556903    4515 generic.go:356] "Generic (PLEG): container finished" podID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerID="b95ac8e55c3f3600b99ad8cf62b8829982aec1bf088ff4d6bde05c86d55ed46b" exitCode=60
Nov 02 08:30:03.559349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.557094    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerDied","Data":"b95ac8e55c3f3600b99ad8cf62b8829982aec1bf088ff4d6bde05c86d55ed46b"}
Nov 02 08:30:03.559349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.557637    4515 scope.go:122] "RemoveContainer" containerID="b95ac8e55c3f3600b99ad8cf62b8829982aec1bf088ff4d6bde05c86d55ed46b"
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.560868    4515 generic.go:356] "Generic (PLEG): container finished" podID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerID="cc4aa4ae74dd704202ae80936516ef56c5ee75d8e2b0b0bdc0d5a3c1ca6e8325" exitCode=60
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.560985    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerDied","Data":"cc4aa4ae74dd704202ae80936516ef56c5ee75d8e2b0b0bdc0d5a3c1ca6e8325"}
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.561514    4515 scope.go:122] "RemoveContainer" containerID="cc4aa4ae74dd704202ae80936516ef56c5ee75d8e2b0b0bdc0d5a3c1ca6e8325"
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.564452    4515 generic.go:356] "Generic (PLEG): container finished" podID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerID="1e0d0c883305929c6086a146adaa62c3b192b68e8001241eb4db023a92b804e7" exitCode=60
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.564551    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerDied","Data":"1e0d0c883305929c6086a146adaa62c3b192b68e8001241eb4db023a92b804e7"}
Nov 02 08:30:03.565367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.565024    4515 scope.go:122] "RemoveContainer" containerID="1e0d0c883305929c6086a146adaa62c3b192b68e8001241eb4db023a92b804e7"
Nov 02 08:30:03.600999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.600836    4515 generic.go:356] "Generic (PLEG): container finished" podID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerID="31fe6546ccc42e6dca4af9a2106e1883dfb33a9a516f0fa366ea4370c126e4a1" exitCode=60
Nov 02 08:30:03.600999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.600929    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerDied","Data":"31fe6546ccc42e6dca4af9a2106e1883dfb33a9a516f0fa366ea4370c126e4a1"}
Nov 02 08:30:03.602526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.602501    4515 scope.go:122] "RemoveContainer" containerID="31fe6546ccc42e6dca4af9a2106e1883dfb33a9a516f0fa366ea4370c126e4a1"
Nov 02 08:30:03.674411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.674375    4515 generic.go:356] "Generic (PLEG): container finished" podID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerID="4b2591797df6378381562341fb32cd179591c2591e5d73c911d5edd0d3f4031a" exitCode=60
Nov 02 08:30:03.674771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.674746    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerDied","Data":"4b2591797df6378381562341fb32cd179591c2591e5d73c911d5edd0d3f4031a"}
Nov 02 08:30:03.678505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.678480    4515 scope.go:122] "RemoveContainer" containerID="4b2591797df6378381562341fb32cd179591c2591e5d73c911d5edd0d3f4031a"
Nov 02 08:30:03.724242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.715258    4515 generic.go:356] "Generic (PLEG): container finished" podID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerID="e4d30eeb1e8cb08ad4789d34df2815efced2df897472ce465742f617634736ba" exitCode=60
Nov 02 08:30:03.724242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.715459    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerDied","Data":"e4d30eeb1e8cb08ad4789d34df2815efced2df897472ce465742f617634736ba"}
Nov 02 08:30:03.724242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.715886    4515 scope.go:122] "RemoveContainer" containerID="e4d30eeb1e8cb08ad4789d34df2815efced2df897472ce465742f617634736ba"
Nov 02 08:30:03.736815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.736773    4515 generic.go:356] "Generic (PLEG): container finished" podID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerID="37167c6059df8c5c701c731d27c66a2a6fe5c03c400c3c161d767b5ffaaf655d" exitCode=60
Nov 02 08:30:03.736969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.736844    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerDied","Data":"37167c6059df8c5c701c731d27c66a2a6fe5c03c400c3c161d767b5ffaaf655d"}
Nov 02 08:30:03.738713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:03.737406    4515 scope.go:122] "RemoveContainer" containerID="37167c6059df8c5c701c731d27c66a2a6fe5c03c400c3c161d767b5ffaaf655d"
Nov 02 08:30:04.746097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.745998    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerStarted","Data":"ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"}
Nov 02 08:30:04.748093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.748062    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerStarted","Data":"da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"}
Nov 02 08:30:04.749950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.749929    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerStarted","Data":"8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"}
Nov 02 08:30:04.751930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.751903    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerStarted","Data":"5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"}
Nov 02 08:30:04.753818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.753793    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerStarted","Data":"5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"}
Nov 02 08:30:04.755786 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.755758    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerStarted","Data":"ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"}
Nov 02 08:30:04.757552 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.757524    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerStarted","Data":"2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"}
Nov 02 08:30:04.761971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.760556    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerStarted","Data":"6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"}
Nov 02 08:30:04.922209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:04.922153    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" podStartSLOduration=4.922135188 podStartE2EDuration="4.922135188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.916464653 +0000 UTC m=+3949.678901563" watchObservedRunningTime="2026-11-02 08:30:04.922135188 +0000 UTC m=+3949.684572094"
Nov 02 08:30:05.050085 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.050038    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" podStartSLOduration=5.05002283 podStartE2EDuration="5.05002283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.04684724 +0000 UTC m=+3949.809284145" watchObservedRunningTime="2026-11-02 08:30:05.05002283 +0000 UTC m=+3949.812459735"
Nov 02 08:30:05.089988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.089929    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" podStartSLOduration=5.089911906 podStartE2EDuration="5.089911906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.087739341 +0000 UTC m=+3949.850176247" watchObservedRunningTime="2026-11-02 08:30:05.089911906 +0000 UTC m=+3949.852348813"
Nov 02 08:30:05.771004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.770970    4515 generic.go:356] "Generic (PLEG): container finished" podID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerID="2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2" exitCode=60
Nov 02 08:30:05.771456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.771051    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerDied","Data":"2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"}
Nov 02 08:30:05.771456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.771149    4515 scope.go:122] "RemoveContainer" containerID="2d68c23572684451a7b74f0631949ed793c16a53e93f70e57948a1a2edd6b402"
Nov 02 08:30:05.771456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.771391    4515 scope.go:122] "RemoveContainer" containerID="2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"
Nov 02 08:30:05.771784 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.771763    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq_odh-ai-gateway-infra(78369e14-b18f-4ad8-a30b-32c1b99b49d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7"
Nov 02 08:30:05.773227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.773207    4515 generic.go:356] "Generic (PLEG): container finished" podID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerID="6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f" exitCode=60
Nov 02 08:30:05.773345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.773290    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerDied","Data":"6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"}
Nov 02 08:30:05.773571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.773555    4515 scope.go:122] "RemoveContainer" containerID="6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"
Nov 02 08:30:05.773917 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.773895    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj_odh-ai-gateway-infra(dd7ea61f-ef56-4699-a8df-a84332efb81e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e"
Nov 02 08:30:05.775297 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.775271    4515 generic.go:356] "Generic (PLEG): container finished" podID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerID="ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48" exitCode=60
Nov 02 08:30:05.775392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.775348    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerDied","Data":"ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"}
Nov 02 08:30:05.775671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.775653    4515 scope.go:122] "RemoveContainer" containerID="ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"
Nov 02 08:30:05.775983 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.775960    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh_odh-ai-gateway-infra(43443cca-aef0-479b-9fd2-c7545cdd676a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a"
Nov 02 08:30:05.777278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.777257    4515 generic.go:356] "Generic (PLEG): container finished" podID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerID="da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f" exitCode=60
Nov 02 08:30:05.777380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.777290    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerDied","Data":"da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"}
Nov 02 08:30:05.777673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.777655    4515 scope.go:122] "RemoveContainer" containerID="da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"
Nov 02 08:30:05.778034 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.778006    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm_odh-ai-gateway-infra(f3596205-93a5-4e51-962b-9e49ccef4aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8"
Nov 02 08:30:05.779352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.779297    4515 generic.go:356] "Generic (PLEG): container finished" podID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerID="8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79" exitCode=60
Nov 02 08:30:05.779465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.779382    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerDied","Data":"8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"}
Nov 02 08:30:05.779667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.779645    4515 scope.go:122] "RemoveContainer" containerID="8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"
Nov 02 08:30:05.779983 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.779959    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j_odh-ai-gateway-infra(9b45d301-a087-40d2-8b4e-4f49130cb648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648"
Nov 02 08:30:05.781376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.781356    4515 generic.go:356] "Generic (PLEG): container finished" podID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerID="5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138" exitCode=60
Nov 02 08:30:05.781467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.781444    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerDied","Data":"5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"}
Nov 02 08:30:05.781656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.781642    4515 scope.go:122] "RemoveContainer" containerID="5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"
Nov 02 08:30:05.781908 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.781889    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk_odh-ai-gateway-infra(d2f5a163-b184-4d6c-839c-9f239200f19a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a"
Nov 02 08:30:05.783379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.783356    4515 generic.go:356] "Generic (PLEG): container finished" podID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerID="5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7" exitCode=60
Nov 02 08:30:05.783469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.783437    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerDied","Data":"5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"}
Nov 02 08:30:05.783639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.783626    4515 scope.go:122] "RemoveContainer" containerID="5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"
Nov 02 08:30:05.783889 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.783870    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893470-nqn8c_odh-ai-gateway-infra(b922b513-40e7-4073-a7a2-aaf7d1dbb081)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081"
Nov 02 08:30:05.785478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.785459    4515 generic.go:356] "Generic (PLEG): container finished" podID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerID="ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969" exitCode=60
Nov 02 08:30:05.785590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.785504    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerDied","Data":"ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"}
Nov 02 08:30:05.785713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.785702    4515 scope.go:122] "RemoveContainer" containerID="ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"
Nov 02 08:30:05.785936 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:05.785921    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf_odh-ai-gateway-infra(ff23e2c0-13a3-4364-b2db-62d7aa327278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278"
Nov 02 08:30:05.786342 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.786296    4515 scope.go:122] "RemoveContainer" containerID="b95ac8e55c3f3600b99ad8cf62b8829982aec1bf088ff4d6bde05c86d55ed46b"
Nov 02 08:30:05.799878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.799862    4515 scope.go:122] "RemoveContainer" containerID="cc4aa4ae74dd704202ae80936516ef56c5ee75d8e2b0b0bdc0d5a3c1ca6e8325"
Nov 02 08:30:05.813704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.813686    4515 scope.go:122] "RemoveContainer" containerID="1e0d0c883305929c6086a146adaa62c3b192b68e8001241eb4db023a92b804e7"
Nov 02 08:30:05.837104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.837078    4515 scope.go:122] "RemoveContainer" containerID="31fe6546ccc42e6dca4af9a2106e1883dfb33a9a516f0fa366ea4370c126e4a1"
Nov 02 08:30:05.852051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.852028    4515 scope.go:122] "RemoveContainer" containerID="4b2591797df6378381562341fb32cd179591c2591e5d73c911d5edd0d3f4031a"
Nov 02 08:30:05.865832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.865811    4515 scope.go:122] "RemoveContainer" containerID="e4d30eeb1e8cb08ad4789d34df2815efced2df897472ce465742f617634736ba"
Nov 02 08:30:05.879249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:05.879229    4515 scope.go:122] "RemoveContainer" containerID="37167c6059df8c5c701c731d27c66a2a6fe5c03c400c3c161d767b5ffaaf655d"
Nov 02 08:30:06.795502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.795468    4515 scope.go:122] "RemoveContainer" containerID="ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"
Nov 02 08:30:06.795981 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.795826    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh_odh-ai-gateway-infra(43443cca-aef0-479b-9fd2-c7545cdd676a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a"
Nov 02 08:30:06.797083 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.797063    4515 scope.go:122] "RemoveContainer" containerID="da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"
Nov 02 08:30:06.797405 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.797378    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm_odh-ai-gateway-infra(f3596205-93a5-4e51-962b-9e49ccef4aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8"
Nov 02 08:30:06.798682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.798664    4515 scope.go:122] "RemoveContainer" containerID="8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"
Nov 02 08:30:06.798978 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.798959    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j_odh-ai-gateway-infra(9b45d301-a087-40d2-8b4e-4f49130cb648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648"
Nov 02 08:30:06.800212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.800193    4515 scope.go:122] "RemoveContainer" containerID="5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"
Nov 02 08:30:06.800522 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.800495    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk_odh-ai-gateway-infra(d2f5a163-b184-4d6c-839c-9f239200f19a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a"
Nov 02 08:30:06.801760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.801742    4515 scope.go:122] "RemoveContainer" containerID="5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"
Nov 02 08:30:06.802012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.801990    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893470-nqn8c_odh-ai-gateway-infra(b922b513-40e7-4073-a7a2-aaf7d1dbb081)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081"
Nov 02 08:30:06.803225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.803205    4515 scope.go:122] "RemoveContainer" containerID="ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"
Nov 02 08:30:06.803458 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.803438    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf_odh-ai-gateway-infra(ff23e2c0-13a3-4364-b2db-62d7aa327278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278"
Nov 02 08:30:06.804818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.804803    4515 scope.go:122] "RemoveContainer" containerID="2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"
Nov 02 08:30:06.805087 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.805064    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq_odh-ai-gateway-infra(78369e14-b18f-4ad8-a30b-32c1b99b49d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7"
Nov 02 08:30:06.806426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:06.806409    4515 scope.go:122] "RemoveContainer" containerID="6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"
Nov 02 08:30:06.806624 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:06.806608    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj_odh-ai-gateway-infra(dd7ea61f-ef56-4699-a8df-a84332efb81e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e"
Nov 02 08:30:08.822140 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:08.822107    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 08:30:13.823178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:13.823132    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing" containerID="cri-o://f91694e4a23b7bca33f5abf7e9e17ea5949bcb7869ffd4d4d0c532e4286863ab" gracePeriod=25
Nov 02 08:30:16.366270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:16.366233    4515 scope.go:122] "RemoveContainer" containerID="5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"
Nov 02 08:30:16.366749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:16.366358    4515 scope.go:122] "RemoveContainer" containerID="5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"
Nov 02 08:30:16.886250 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:16.886148    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerStarted","Data":"d4b24cf777dd85d0da2b7cbbc02a36f1569ab8ee9085279027310b3217964065"}
Nov 02 08:30:16.888191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:16.888162    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerStarted","Data":"7a51369b007c3ddc3c7c496b0a79df97a0a74cb8e2b6e56ce025da8e9e317cf0"}
Nov 02 08:30:17.366721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.366688    4515 scope.go:122] "RemoveContainer" containerID="6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"
Nov 02 08:30:17.367227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.366800    4515 scope.go:122] "RemoveContainer" containerID="da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"
Nov 02 08:30:17.730585 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:17.730531    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:30:17.902295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.902193    4515 generic.go:356] "Generic (PLEG): container finished" podID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerID="7a51369b007c3ddc3c7c496b0a79df97a0a74cb8e2b6e56ce025da8e9e317cf0" exitCode=60
Nov 02 08:30:17.902295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.902221    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerDied","Data":"7a51369b007c3ddc3c7c496b0a79df97a0a74cb8e2b6e56ce025da8e9e317cf0"}
Nov 02 08:30:17.902295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.902285    4515 scope.go:122] "RemoveContainer" containerID="5becb8aa35d11cb5d39aef3b80e0d1b53cbe555902c3b60fc28199f6ab9421a7"
Nov 02 08:30:17.902798 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.902769    4515 scope.go:122] "RemoveContainer" containerID="7a51369b007c3ddc3c7c496b0a79df97a0a74cb8e2b6e56ce025da8e9e317cf0"
Nov 02 08:30:17.903223 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:17.903192    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893470-nqn8c_odh-ai-gateway-infra(b922b513-40e7-4073-a7a2-aaf7d1dbb081)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081"
Nov 02 08:30:17.905242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.905214    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerStarted","Data":"99034355825a39f6bf77b62371812510e890093d885f65941b161dfe88974f22"}
Nov 02 08:30:17.908162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.908140    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerStarted","Data":"e28301944fec136e4c31ad49a53470abfb221efb18f0d6a5b82a97e8cdc3daa0"}
Nov 02 08:30:17.911222 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.911195    4515 generic.go:356] "Generic (PLEG): container finished" podID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerID="d4b24cf777dd85d0da2b7cbbc02a36f1569ab8ee9085279027310b3217964065" exitCode=60
Nov 02 08:30:17.911332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.911270    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerDied","Data":"d4b24cf777dd85d0da2b7cbbc02a36f1569ab8ee9085279027310b3217964065"}
Nov 02 08:30:17.911686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.911668    4515 scope.go:122] "RemoveContainer" containerID="d4b24cf777dd85d0da2b7cbbc02a36f1569ab8ee9085279027310b3217964065"
Nov 02 08:30:17.911995 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:17.911974    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk_odh-ai-gateway-infra(d2f5a163-b184-4d6c-839c-9f239200f19a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a"
Nov 02 08:30:17.919973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.919956    4515 scope.go:122] "RemoveContainer" containerID="5a1b4f1238137139dac319269dff6be71a209cd7483c945adebfd86a77967138"
Nov 02 08:30:17.939443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.939417    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:17.993847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:17.993810    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:18.366131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.366101    4515 scope.go:122] "RemoveContainer" containerID="8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"
Nov 02 08:30:18.920426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.920332    4515 generic.go:356] "Generic (PLEG): container finished" podID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerID="99034355825a39f6bf77b62371812510e890093d885f65941b161dfe88974f22" exitCode=60
Nov 02 08:30:18.920426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.920391    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerDied","Data":"99034355825a39f6bf77b62371812510e890093d885f65941b161dfe88974f22"}
Nov 02 08:30:18.921091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.920462    4515 scope.go:122] "RemoveContainer" containerID="6d550ef792e3060eea87c19ff74aaaf66375095497199c3603bff40617a7783f"
Nov 02 08:30:18.921091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.920894    4515 scope.go:122] "RemoveContainer" containerID="99034355825a39f6bf77b62371812510e890093d885f65941b161dfe88974f22"
Nov 02 08:30:18.921278 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:18.921246    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj_odh-ai-gateway-infra(dd7ea61f-ef56-4699-a8df-a84332efb81e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e"
Nov 02 08:30:18.922761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.922736    4515 generic.go:356] "Generic (PLEG): container finished" podID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerID="e28301944fec136e4c31ad49a53470abfb221efb18f0d6a5b82a97e8cdc3daa0" exitCode=60
Nov 02 08:30:18.922860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.922788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerDied","Data":"e28301944fec136e4c31ad49a53470abfb221efb18f0d6a5b82a97e8cdc3daa0"}
Nov 02 08:30:18.923248 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.923222    4515 scope.go:122] "RemoveContainer" containerID="e28301944fec136e4c31ad49a53470abfb221efb18f0d6a5b82a97e8cdc3daa0"
Nov 02 08:30:18.923645 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:18.923623    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm_odh-ai-gateway-infra(f3596205-93a5-4e51-962b-9e49ccef4aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8"
Nov 02 08:30:18.924852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.924825    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerStarted","Data":"4d86b70ee32cda490a4c4c839c12664995f64e7bc5f3f41cafa44fb5724afb4e"}
Nov 02 08:30:18.936153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:18.936126    4515 scope.go:122] "RemoveContainer" containerID="da72e753b97f24bb49f98d90106800c2548cbb006101ba96a465f012da60844f"
Nov 02 08:30:19.042181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.041999    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:19.083814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.083785    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:19.125202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.123707    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:19.172270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.172210    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:19.254415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.254382    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\" (UniqueName: \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\") pod \"d2f5a163-b184-4d6c-839c-9f239200f19a\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") "
Nov 02 08:30:19.254625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.254514    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\") pod \"d2f5a163-b184-4d6c-839c-9f239200f19a\" (UID: \"d2f5a163-b184-4d6c-839c-9f239200f19a\") "
Nov 02 08:30:19.254625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.254550    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\") pod \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") "
Nov 02 08:30:19.254625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.254614    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\" (UniqueName: \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\") pod \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\" (UID: \"b922b513-40e7-4073-a7a2-aaf7d1dbb081\") "
Nov 02 08:30:19.254914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.254888    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca" pod "d2f5a163-b184-4d6c-839c-9f239200f19a" (UID: "d2f5a163-b184-4d6c-839c-9f239200f19a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:19.255064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.255037    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca" pod "b922b513-40e7-4073-a7a2-aaf7d1dbb081" (UID: "b922b513-40e7-4073-a7a2-aaf7d1dbb081"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:19.255320 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.255286    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f5a163-b184-4d6c-839c-9f239200f19a-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:19.255420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.255332    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922b513-40e7-4073-a7a2-aaf7d1dbb081-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:19.317397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.316891    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n" pod "b922b513-40e7-4073-a7a2-aaf7d1dbb081" (UID: "b922b513-40e7-4073-a7a2-aaf7d1dbb081"). InnerVolumeSpecName "kube-api-access-mp28n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:19.317397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.316999    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4" pod "d2f5a163-b184-4d6c-839c-9f239200f19a" (UID: "d2f5a163-b184-4d6c-839c-9f239200f19a"). InnerVolumeSpecName "kube-api-access-mfjg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:19.357086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.357049    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp28n\" (UniqueName: \"kubernetes.io/projected/b922b513-40e7-4073-a7a2-aaf7d1dbb081-kube-api-access-mp28n\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:19.357086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.357082    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfjg4\" (UniqueName: \"kubernetes.io/projected/d2f5a163-b184-4d6c-839c-9f239200f19a-kube-api-access-mfjg4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:19.366249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.366222    4515 scope.go:122] "RemoveContainer" containerID="2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"
Nov 02 08:30:19.366436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.366378    4515 scope.go:122] "RemoveContainer" containerID="ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"
Nov 02 08:30:19.366524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.366505    4515 scope.go:122] "RemoveContainer" containerID="ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"
Nov 02 08:30:19.465002 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:19.464956    4515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f5a163_b184_4d6c_839c_9f239200f19a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb922b513_40e7_4073_a7a2_aaf7d1dbb081.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb922b513_40e7_4073_a7a2_aaf7d1dbb081.slice/crio-1cc90c977f74197dd11b9796a352f394e4e15c57e4d189ad4ed597a1b5267060\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f5a163_b184_4d6c_839c_9f239200f19a.slice/crio-f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:19.938461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.938431    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerStarted","Data":"5ce7d39e326d2553ca6cea2d477171dd7cc6e2cdb4854ac9042abd2fb6256505"}
Nov 02 08:30:19.942409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.942378    4515 generic.go:356] "Generic (PLEG): container finished" podID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerID="4d86b70ee32cda490a4c4c839c12664995f64e7bc5f3f41cafa44fb5724afb4e" exitCode=60
Nov 02 08:30:19.942559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.942451    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerDied","Data":"4d86b70ee32cda490a4c4c839c12664995f64e7bc5f3f41cafa44fb5724afb4e"}
Nov 02 08:30:19.942559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.942513    4515 scope.go:122] "RemoveContainer" containerID="8c9b86dec27aeb1736e4a452e11b20693d33047c67de256407230e9c9fb88f79"
Nov 02 08:30:19.942962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.942940    4515 scope.go:122] "RemoveContainer" containerID="4d86b70ee32cda490a4c4c839c12664995f64e7bc5f3f41cafa44fb5724afb4e"
Nov 02 08:30:19.943355 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:19.943326    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j_odh-ai-gateway-infra(9b45d301-a087-40d2-8b4e-4f49130cb648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648"
Nov 02 08:30:19.943933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.943917    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"
Nov 02 08:30:19.944013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.943962    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk" event={"ID":"d2f5a163-b184-4d6c-839c-9f239200f19a","Type":"ContainerDied","Data":"f04ab767ef0fcf3821b055699efddd3878e43645b656c0584d230d82a1d86c79"}
Nov 02 08:30:19.945506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.945490    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"
Nov 02 08:30:19.945631 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.945518    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c" event={"ID":"b922b513-40e7-4073-a7a2-aaf7d1dbb081","Type":"ContainerDied","Data":"1cc90c977f74197dd11b9796a352f394e4e15c57e4d189ad4ed597a1b5267060"}
Nov 02 08:30:19.947860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.947837    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerStarted","Data":"a195cfac53571df5dbf40ae958fcf44dd6cfe898e25b090bfe6db85711e590b9"}
Nov 02 08:30:19.950256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.950233    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerStarted","Data":"cb640072e39b1b0f49f25d7f56acc1f995f7b50f4d33d44db17e50b3f041e33a"}
Nov 02 08:30:19.957986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.957965    4515 scope.go:122] "RemoveContainer" containerID="d4b24cf777dd85d0da2b7cbbc02a36f1569ab8ee9085279027310b3217964065"
Nov 02 08:30:19.969183 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:19.969154    4515 scope.go:122] "RemoveContainer" containerID="7a51369b007c3ddc3c7c496b0a79df97a0a74cb8e2b6e56ce025da8e9e317cf0"
Nov 02 08:30:20.021351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.020855    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:20.248471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.248396    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:20.251619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.251583    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:20.327446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.327409    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:20.338285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.338254    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893470-422hk"]
Nov 02 08:30:20.372636 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.372603    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\") pod \"f3596205-93a5-4e51-962b-9e49ccef4aa8\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") "
Nov 02 08:30:20.372838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.372818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\" (UniqueName: \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\") pod \"dd7ea61f-ef56-4699-a8df-a84332efb81e\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") "
Nov 02 08:30:20.372945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.372927    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\" (UniqueName: \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\") pod \"f3596205-93a5-4e51-962b-9e49ccef4aa8\" (UID: \"f3596205-93a5-4e51-962b-9e49ccef4aa8\") "
Nov 02 08:30:20.373031 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.373008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca" pod "f3596205-93a5-4e51-962b-9e49ccef4aa8" (UID: "f3596205-93a5-4e51-962b-9e49ccef4aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:20.373093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.373045    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\") pod \"dd7ea61f-ef56-4699-a8df-a84332efb81e\" (UID: \"dd7ea61f-ef56-4699-a8df-a84332efb81e\") "
Nov 02 08:30:20.373485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.373460    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca" pod "dd7ea61f-ef56-4699-a8df-a84332efb81e" (UID: "dd7ea61f-ef56-4699-a8df-a84332efb81e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:20.373712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.373696    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7ea61f-ef56-4699-a8df-a84332efb81e-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.373790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.373713    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3596205-93a5-4e51-962b-9e49ccef4aa8-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.376477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.376454    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:20.392461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.392438    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893470-nqn8c"]
Nov 02 08:30:20.417271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.416979    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6" pod "f3596205-93a5-4e51-962b-9e49ccef4aa8" (UID: "f3596205-93a5-4e51-962b-9e49ccef4aa8"). InnerVolumeSpecName "kube-api-access-tplb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:20.417666 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.417633    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md" pod "dd7ea61f-ef56-4699-a8df-a84332efb81e" (UID: "dd7ea61f-ef56-4699-a8df-a84332efb81e"). InnerVolumeSpecName "kube-api-access-wd4md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:20.474999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.474918    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4md\" (UniqueName: \"kubernetes.io/projected/dd7ea61f-ef56-4699-a8df-a84332efb81e-kube-api-access-wd4md\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.474999 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.474955    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplb6\" (UniqueName: \"kubernetes.io/projected/f3596205-93a5-4e51-962b-9e49ccef4aa8-kube-api-access-tplb6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.746382 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.746339    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:20.961280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.961196    4515 generic.go:356] "Generic (PLEG): container finished" podID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerID="5ce7d39e326d2553ca6cea2d477171dd7cc6e2cdb4854ac9042abd2fb6256505" exitCode=60
Nov 02 08:30:20.961280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.961265    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerDied","Data":"5ce7d39e326d2553ca6cea2d477171dd7cc6e2cdb4854ac9042abd2fb6256505"}
Nov 02 08:30:20.961863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.961364    4515 scope.go:122] "RemoveContainer" containerID="ade3f0dd0a4fce61949ecc6081ca39707294e4914cb5306429566c4e690dbc48"
Nov 02 08:30:20.961863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.961817    4515 scope.go:122] "RemoveContainer" containerID="5ce7d39e326d2553ca6cea2d477171dd7cc6e2cdb4854ac9042abd2fb6256505"
Nov 02 08:30:20.962199 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:20.962176    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh_odh-ai-gateway-infra(43443cca-aef0-479b-9fd2-c7545cdd676a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a"
Nov 02 08:30:20.962891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.962871    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"
Nov 02 08:30:20.962891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.962884    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm" event={"ID":"f3596205-93a5-4e51-962b-9e49ccef4aa8","Type":"ContainerDied","Data":"17356bd456e46abd6ef584c9d1927de1ac331770a2b887ceb15dcc3d8b38b778"}
Nov 02 08:30:20.968892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.968870    4515 generic.go:356] "Generic (PLEG): container finished" podID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerID="a195cfac53571df5dbf40ae958fcf44dd6cfe898e25b090bfe6db85711e590b9" exitCode=60
Nov 02 08:30:20.968994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.968943    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerDied","Data":"a195cfac53571df5dbf40ae958fcf44dd6cfe898e25b090bfe6db85711e590b9"}
Nov 02 08:30:20.969522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.969386    4515 scope.go:122] "RemoveContainer" containerID="a195cfac53571df5dbf40ae958fcf44dd6cfe898e25b090bfe6db85711e590b9"
Nov 02 08:30:20.969730 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:20.969707    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf_odh-ai-gateway-infra(ff23e2c0-13a3-4364-b2db-62d7aa327278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278"
Nov 02 08:30:20.971103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.971084    4515 generic.go:356] "Generic (PLEG): container finished" podID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerID="cb640072e39b1b0f49f25d7f56acc1f995f7b50f4d33d44db17e50b3f041e33a" exitCode=60
Nov 02 08:30:20.971204 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.971160    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerDied","Data":"cb640072e39b1b0f49f25d7f56acc1f995f7b50f4d33d44db17e50b3f041e33a"}
Nov 02 08:30:20.971588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.971571    4515 scope.go:122] "RemoveContainer" containerID="cb640072e39b1b0f49f25d7f56acc1f995f7b50f4d33d44db17e50b3f041e33a"
Nov 02 08:30:20.971858 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:20.971835    4515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq_odh-ai-gateway-infra(78369e14-b18f-4ad8-a30b-32c1b99b49d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7"
Nov 02 08:30:20.972701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.972677    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj" event={"ID":"dd7ea61f-ef56-4699-a8df-a84332efb81e","Type":"ContainerDied","Data":"97ca66b038487aac517d84eb89279de0a7df0b6eeb7b661f1eaa1d29cce630ce"}
Nov 02 08:30:20.972824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.972779    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"
Nov 02 08:30:20.976947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.976929    4515 scope.go:122] "RemoveContainer" containerID="e28301944fec136e4c31ad49a53470abfb221efb18f0d6a5b82a97e8cdc3daa0"
Nov 02 08:30:20.988541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.988521    4515 scope.go:122] "RemoveContainer" containerID="ff5bfe69a6492cec308e831ca7fabfd2cef4da34056e293cb80f1af19b5aa969"
Nov 02 08:30:20.995995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:20.995969    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:21.006841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.006818    4515 scope.go:122] "RemoveContainer" containerID="2b768b391b4466664e193300491928d9a4beb65c82baa9463a97da9457e15bf2"
Nov 02 08:30:21.033810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.033764    4515 scope.go:122] "RemoveContainer" containerID="99034355825a39f6bf77b62371812510e890093d885f65941b161dfe88974f22"
Nov 02 08:30:21.059874 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.059843    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:21.067506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.067471    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:21.071491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.071461    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893470-p7wtm"]
Nov 02 08:30:21.144405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.144374    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:21.176239 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.176201    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:21.179799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.179772    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893470-tpfgj"]
Nov 02 08:30:21.188380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.188343    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:21.287058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.287021    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\" (UniqueName: \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\") pod \"9b45d301-a087-40d2-8b4e-4f49130cb648\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") "
Nov 02 08:30:21.287245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.287177    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\") pod \"9b45d301-a087-40d2-8b4e-4f49130cb648\" (UID: \"9b45d301-a087-40d2-8b4e-4f49130cb648\") "
Nov 02 08:30:21.287571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.287543    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca" pod "9b45d301-a087-40d2-8b4e-4f49130cb648" (UID: "9b45d301-a087-40d2-8b4e-4f49130cb648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:21.287844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.287820    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b45d301-a087-40d2-8b4e-4f49130cb648-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:21.289399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.289370    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c" pod "9b45d301-a087-40d2-8b4e-4f49130cb648" (UID: "9b45d301-a087-40d2-8b4e-4f49130cb648"). InnerVolumeSpecName "kube-api-access-xzj8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:21.375257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.375221    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" path="/var/lib/kubelet/pods/b922b513-40e7-4073-a7a2-aaf7d1dbb081/volumes"
Nov 02 08:30:21.375767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.375752    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" path="/var/lib/kubelet/pods/d2f5a163-b184-4d6c-839c-9f239200f19a/volumes"
Nov 02 08:30:21.376148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.376135    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" path="/var/lib/kubelet/pods/dd7ea61f-ef56-4699-a8df-a84332efb81e/volumes"
Nov 02 08:30:21.376873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.376857    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" path="/var/lib/kubelet/pods/f3596205-93a5-4e51-962b-9e49ccef4aa8/volumes"
Nov 02 08:30:21.389648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.389622    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj8c\" (UniqueName: \"kubernetes.io/projected/9b45d301-a087-40d2-8b4e-4f49130cb648-kube-api-access-xzj8c\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:21.988627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.988596    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"
Nov 02 08:30:21.989055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.988621    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j" event={"ID":"9b45d301-a087-40d2-8b4e-4f49130cb648","Type":"ContainerDied","Data":"9b38adfc773e76e9a455c186070c755e6af2d3b3026c3bc443336de2f54d14f8"}
Nov 02 08:30:21.989055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:21.988693    4515 scope.go:122] "RemoveContainer" containerID="4d86b70ee32cda490a4c4c839c12664995f64e7bc5f3f41cafa44fb5724afb4e"
Nov 02 08:30:22.076564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.076495    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:22.080908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.079094    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893470-2gw9j"]
Nov 02 08:30:22.188082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.188049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:30:22.270217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.270183    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:22.297323 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.297282    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:22.300830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.300809    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:22.421793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.421762    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\") pod \"43443cca-aef0-479b-9fd2-c7545cdd676a\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") "
Nov 02 08:30:22.422003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.421839    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\" (UniqueName: \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\") pod \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") "
Nov 02 08:30:22.422003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.421872    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\") pod \"ff23e2c0-13a3-4364-b2db-62d7aa327278\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") "
Nov 02 08:30:22.422131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.422033    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\" (UniqueName: \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\") pod \"43443cca-aef0-479b-9fd2-c7545cdd676a\" (UID: \"43443cca-aef0-479b-9fd2-c7545cdd676a\") "
Nov 02 08:30:22.422131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.422121    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\" (UniqueName: \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\") pod \"ff23e2c0-13a3-4364-b2db-62d7aa327278\" (UID: \"ff23e2c0-13a3-4364-b2db-62d7aa327278\") "
Nov 02 08:30:22.422235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.422216    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\") pod \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\" (UID: \"78369e14-b18f-4ad8-a30b-32c1b99b49d7\") "
Nov 02 08:30:22.422292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.422231    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca" pod "43443cca-aef0-479b-9fd2-c7545cdd676a" (UID: "43443cca-aef0-479b-9fd2-c7545cdd676a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.422671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.422644    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca" pod "ff23e2c0-13a3-4364-b2db-62d7aa327278" (UID: "ff23e2c0-13a3-4364-b2db-62d7aa327278"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.423057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.423029    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43443cca-aef0-479b-9fd2-c7545cdd676a-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.423161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.423063    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff23e2c0-13a3-4364-b2db-62d7aa327278-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.423161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.423102    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca" pod "78369e14-b18f-4ad8-a30b-32c1b99b49d7" (UID: "78369e14-b18f-4ad8-a30b-32c1b99b49d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.424655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.424627    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6" pod "ff23e2c0-13a3-4364-b2db-62d7aa327278" (UID: "ff23e2c0-13a3-4364-b2db-62d7aa327278"). InnerVolumeSpecName "kube-api-access-89qh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.424803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.424778    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2" pod "78369e14-b18f-4ad8-a30b-32c1b99b49d7" (UID: "78369e14-b18f-4ad8-a30b-32c1b99b49d7"). InnerVolumeSpecName "kube-api-access-d6jj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.425049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.425022    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv" pod "43443cca-aef0-479b-9fd2-c7545cdd676a" (UID: "43443cca-aef0-479b-9fd2-c7545cdd676a"). InnerVolumeSpecName "kube-api-access-n6mxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.524522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.524444    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6jj2\" (UniqueName: \"kubernetes.io/projected/78369e14-b18f-4ad8-a30b-32c1b99b49d7-kube-api-access-d6jj2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.524522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.524474    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6mxv\" (UniqueName: \"kubernetes.io/projected/43443cca-aef0-479b-9fd2-c7545cdd676a-kube-api-access-n6mxv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.524522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.524484    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qh6\" (UniqueName: \"kubernetes.io/projected/ff23e2c0-13a3-4364-b2db-62d7aa327278-kube-api-access-89qh6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.524522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:22.524494    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78369e14-b18f-4ad8-a30b-32c1b99b49d7-maas-api-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.001996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.001964    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"
Nov 02 08:30:23.001996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.001967    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf" event={"ID":"ff23e2c0-13a3-4364-b2db-62d7aa327278","Type":"ContainerDied","Data":"6777ecf6ef264d5ae7e070f248a460feeaad5688844c327d4027c5e97f02b0f7"}
Nov 02 08:30:23.002584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.002054    4515 scope.go:122] "RemoveContainer" containerID="a195cfac53571df5dbf40ae958fcf44dd6cfe898e25b090bfe6db85711e590b9"
Nov 02 08:30:23.007348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.004682    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"
Nov 02 08:30:23.007348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.004675    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq" event={"ID":"78369e14-b18f-4ad8-a30b-32c1b99b49d7","Type":"ContainerDied","Data":"828a22cd06d19fed4f144b4072bf062148f0d6328a617d297546e03b920e5c1a"}
Nov 02 08:30:23.011665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.011628    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh" event={"ID":"43443cca-aef0-479b-9fd2-c7545cdd676a","Type":"ContainerDied","Data":"ec2975ea8593af29428a0f7599608dd0ab1f639e110b3c25a481e00a0e67bf48"}
Nov 02 08:30:23.012234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.012213    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"
Nov 02 08:30:23.016097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.016076    4515 scope.go:122] "RemoveContainer" containerID="cb640072e39b1b0f49f25d7f56acc1f995f7b50f4d33d44db17e50b3f041e33a"
Nov 02 08:30:23.026443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.026424    4515 scope.go:122] "RemoveContainer" containerID="5ce7d39e326d2553ca6cea2d477171dd7cc6e2cdb4854ac9042abd2fb6256505"
Nov 02 08:30:23.071094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.071056    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:23.084549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.084512    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893470-9fbvq"]
Nov 02 08:30:23.116429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.116396    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:23.119873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.119838    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4b85d50c-29893470-rnhlf"]
Nov 02 08:30:23.154294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.154258    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:23.160868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.160834    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893470-wrwvh"]
Nov 02 08:30:23.384138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.384002    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" path="/var/lib/kubelet/pods/43443cca-aef0-479b-9fd2-c7545cdd676a/volumes"
Nov 02 08:30:23.384661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.384641    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" path="/var/lib/kubelet/pods/78369e14-b18f-4ad8-a30b-32c1b99b49d7/volumes"
Nov 02 08:30:23.385118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.385103    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" path="/var/lib/kubelet/pods/9b45d301-a087-40d2-8b4e-4f49130cb648/volumes"
Nov 02 08:30:23.385779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:23.385759    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" path="/var/lib/kubelet/pods/ff23e2c0-13a3-4364-b2db-62d7aa327278/volumes"
Nov 02 08:30:30.746503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:30.746463    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:34.118129 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.118091    4515 generic.go:356] "Generic (PLEG): container finished" podID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerID="f91694e4a23b7bca33f5abf7e9e17ea5949bcb7869ffd4d4d0c532e4286863ab" exitCode=0
Nov 02 08:30:34.118492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.118172    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" event={"ID":"b11c1b49-75dd-4ee4-ba46-af5a755cb572","Type":"ContainerDied","Data":"f91694e4a23b7bca33f5abf7e9e17ea5949bcb7869ffd4d4d0c532e4286863ab"}
Nov 02 08:30:34.147533 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.147497    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 08:30:34.294175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.294121    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\") pod \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") "
Nov 02 08:30:34.294405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.294270    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\" (UniqueName: \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\") pod \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\" (UID: \"b11c1b49-75dd-4ee4-ba46-af5a755cb572\") "
Nov 02 08:30:34.294538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.294512    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume" pod "b11c1b49-75dd-4ee4-ba46-af5a755cb572" (UID: "b11c1b49-75dd-4ee4-ba46-af5a755cb572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:34.294871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.294849    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11c1b49-75dd-4ee4-ba46-af5a755cb572-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:34.296498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.296473    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv" pod "b11c1b49-75dd-4ee4-ba46-af5a755cb572" (UID: "b11c1b49-75dd-4ee4-ba46-af5a755cb572"). InnerVolumeSpecName "kube-api-access-vkgnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:34.395970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:34.395889    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgnv\" (UniqueName: \"kubernetes.io/projected/b11c1b49-75dd-4ee4-ba46-af5a755cb572-kube-api-access-vkgnv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:35.126756 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.126723    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"
Nov 02 08:30:35.127220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.126749    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp" event={"ID":"b11c1b49-75dd-4ee4-ba46-af5a755cb572","Type":"ContainerDied","Data":"fa9aecbceb138e4a3626f7e6e47f10119b52900e0c2fb1fcc205fe8a83edb41d"}
Nov 02 08:30:35.127220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.126818    4515 scope.go:122] "RemoveContainer" containerID="f91694e4a23b7bca33f5abf7e9e17ea5949bcb7869ffd4d4d0c532e4286863ab"
Nov 02 08:30:35.180043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.179998    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 08:30:35.192226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.192195    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-32ea65-798cd6f665-l6xzp"]
Nov 02 08:30:35.374956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:35.374924    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" path="/var/lib/kubelet/pods/b11c1b49-75dd-4ee4-ba46-af5a755cb572/volumes"
Nov 02 08:30:42.313301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.313259    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 08:30:42.313816 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.313514    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" podUID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" containerName="maas-api" containerID="cri-o://836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56" gracePeriod=30
Nov 02 08:30:42.643334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.642041    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 08:30:42.722166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.722130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\" (UniqueName: \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\") pod \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") "
Nov 02 08:30:42.722405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.722225    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\") pod \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") "
Nov 02 08:30:42.722405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.722262    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\") pod \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") "
Nov 02 08:30:42.722544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.722442    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\") pod \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\" (UID: \"5c3a587a-54db-4b3a-b2cb-62dbdcf91179\") "
Nov 02 08:30:42.722708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.722677    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca" pod "5c3a587a-54db-4b3a-b2cb-62dbdcf91179" (UID: "5c3a587a-54db-4b3a-b2cb-62dbdcf91179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:42.723104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.723080    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:42.724716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.724692    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr" pod "5c3a587a-54db-4b3a-b2cb-62dbdcf91179" (UID: "5c3a587a-54db-4b3a-b2cb-62dbdcf91179"). InnerVolumeSpecName "kube-api-access-5nlkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:42.724961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.724936    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls" pod "5c3a587a-54db-4b3a-b2cb-62dbdcf91179" (UID: "5c3a587a-54db-4b3a-b2cb-62dbdcf91179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:30:42.725121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.725104    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs" pod "5c3a587a-54db-4b3a-b2cb-62dbdcf91179" (UID: "5c3a587a-54db-4b3a-b2cb-62dbdcf91179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:30:42.857490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.839573    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:42.857490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.839611    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:42.857490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:42.839626    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlkr\" (UniqueName: \"kubernetes.io/projected/5c3a587a-54db-4b3a-b2cb-62dbdcf91179-kube-api-access-5nlkr\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:43.187650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.187558    4515 generic.go:356] "Generic (PLEG): container finished" podID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" containerID="836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56" exitCode=0
Nov 02 08:30:43.187650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.187613    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"
Nov 02 08:30:43.187917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.187604    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" event={"ID":"5c3a587a-54db-4b3a-b2cb-62dbdcf91179","Type":"ContainerDied","Data":"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"}
Nov 02 08:30:43.187917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.187749    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262" event={"ID":"5c3a587a-54db-4b3a-b2cb-62dbdcf91179","Type":"ContainerDied","Data":"5f068459367b99fcf37119499832644eb9cecbd0255673b9021ad53589a27fd8"}
Nov 02 08:30:43.187917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.187782    4515 scope.go:122] "RemoveContainer" containerID="836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"
Nov 02 08:30:43.199780 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.199761    4515 scope.go:122] "RemoveContainer" containerID="836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"
Nov 02 08:30:43.200081 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:43.200058    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56\": container with ID starting with 836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56 not found: ID does not exist" containerID="836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"
Nov 02 08:30:43.200190 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.200092    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56"} err="failed to get container status \"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56\": rpc error: code = NotFound desc = could not find container \"836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56\": container with ID starting with 836d6a922631a80f89c7bd731277be0d2611cf3550ac113b26bc9037c8730b56 not found: ID does not exist"
Nov 02 08:30:43.240240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.240208    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 08:30:43.252218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.252188    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-32ea65-5c6f7f8677-ml262"]
Nov 02 08:30:43.374546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:43.374511    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" path="/var/lib/kubelet/pods/5c3a587a-54db-4b3a-b2cb-62dbdcf91179/volumes"
Nov 02 08:30:47.723109 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:47.723070    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:30:52.833256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:52.833216    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:30:52.985485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:52.985438    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:30:57.834532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:57.834434    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" podUID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" containerName="payload-processing" containerID="cri-o://cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba" gracePeriod=25
Nov 02 08:30:57.986431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:57.986360    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" podUID="98fba58e-785e-4418-a949-d99701bff54f" containerName="payload-pre-processing" containerID="cri-o://f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991" gracePeriod=25
Nov 02 08:30:58.135288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.135260    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:30:58.263247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.263218    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\" (UniqueName: \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\") pod \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") "
Nov 02 08:30:58.263405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.263395    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\") pod \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\" (UID: \"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6\") "
Nov 02 08:30:58.263835 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.263809    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume" pod "c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" (UID: "c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:58.265538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.265514    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht" pod "c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" (UID: "c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6"). InnerVolumeSpecName "kube-api-access-sjnht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:58.267258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.267242    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:30:58.292735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.292692    4515 generic.go:356] "Generic (PLEG): container finished" podID="98fba58e-785e-4418-a949-d99701bff54f" containerID="f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991" exitCode=0
Nov 02 08:30:58.292897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.292739    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"
Nov 02 08:30:58.292897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.292773    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" event={"ID":"98fba58e-785e-4418-a949-d99701bff54f","Type":"ContainerDied","Data":"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"}
Nov 02 08:30:58.292897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.292852    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk" event={"ID":"98fba58e-785e-4418-a949-d99701bff54f","Type":"ContainerDied","Data":"bede0c35f773011219f95af339711e7e887b49712d94c29961e2a4fc6122b8f5"}
Nov 02 08:30:58.292897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.292874    4515 scope.go:122] "RemoveContainer" containerID="f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"
Nov 02 08:30:58.294300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.294274    4515 generic.go:356] "Generic (PLEG): container finished" podID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" containerID="cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba" exitCode=0
Nov 02 08:30:58.294433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.294335    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" event={"ID":"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6","Type":"ContainerDied","Data":"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"}
Nov 02 08:30:58.294433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.294348    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"
Nov 02 08:30:58.294433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.294382    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t" event={"ID":"c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6","Type":"ContainerDied","Data":"b8b063740460670578554b6440d1636dbdca072151eee60ca371111fc201548b"}
Nov 02 08:30:58.303340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.303325    4515 scope.go:122] "RemoveContainer" containerID="f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"
Nov 02 08:30:58.303617 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:58.303597    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991\": container with ID starting with f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991 not found: ID does not exist" containerID="f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"
Nov 02 08:30:58.303671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.303625    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991"} err="failed to get container status \"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991\": rpc error: code = NotFound desc = could not find container \"f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991\": container with ID starting with f639815c1bc768105a2453dff4481ecb60570727682bcc106c9d325d5d3c3991 not found: ID does not exist"
Nov 02 08:30:58.303671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.303644    4515 scope.go:122] "RemoveContainer" containerID="cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"
Nov 02 08:30:58.316584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.316569    4515 scope.go:122] "RemoveContainer" containerID="cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"
Nov 02 08:30:58.316981 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:30:58.316951    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba\": container with ID starting with cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba not found: ID does not exist" containerID="cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"
Nov 02 08:30:58.317060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.316988    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba"} err="failed to get container status \"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba\": rpc error: code = NotFound desc = could not find container \"cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba\": container with ID starting with cbfd369ba453a1935bf6f66cee9f7963fcabf7aa3a087af385260c6483efcfba not found: ID does not exist"
Nov 02 08:30:58.357690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.357610    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:30:58.363372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.363339    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4b85d50c-5ccbc9dfd5-8gb8t"]
Nov 02 08:30:58.364543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.364523    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\") pod \"98fba58e-785e-4418-a949-d99701bff54f\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") "
Nov 02 08:30:58.364722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.364710    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\" (UniqueName: \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\") pod \"98fba58e-785e-4418-a949-d99701bff54f\" (UID: \"98fba58e-785e-4418-a949-d99701bff54f\") "
Nov 02 08:30:58.364902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.364881    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume" pod "98fba58e-785e-4418-a949-d99701bff54f" (UID: "98fba58e-785e-4418-a949-d99701bff54f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:58.365201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.365169    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fba58e-785e-4418-a949-d99701bff54f-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.365201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.365201    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.365387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.365211    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnht\" (UniqueName: \"kubernetes.io/projected/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6-kube-api-access-sjnht\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.366786 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.366767    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25" pod "98fba58e-785e-4418-a949-d99701bff54f" (UID: "98fba58e-785e-4418-a949-d99701bff54f"). InnerVolumeSpecName "kube-api-access-k9d25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:58.466972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.466940    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d25\" (UniqueName: \"kubernetes.io/projected/98fba58e-785e-4418-a949-d99701bff54f-kube-api-access-k9d25\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.640122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.640034    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:30:58.649343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:58.649296    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b85d50c-84b44fb699-nh9kk"]
Nov 02 08:30:59.375043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:59.375008    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fba58e-785e-4418-a949-d99701bff54f" path="/var/lib/kubelet/pods/98fba58e-785e-4418-a949-d99701bff54f/volumes"
Nov 02 08:30:59.375511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:30:59.375496    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" path="/var/lib/kubelet/pods/c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6/volumes"
Nov 02 08:31:00.413889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:00.413853    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 08:31:03.316837 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:03.316797    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:31:03.317399 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:03.317103    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy" containerID="cri-o://cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d" gracePeriod=30
Nov 02 08:31:05.414702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:05.414660    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing" containerID="cri-o://baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b" gracePeriod=25
Nov 02 08:31:05.987945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:05.987901    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:08.610209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.610182    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:31:08.720279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.720242    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.720995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.720877    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:08.721161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721031    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721136    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721237    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721272    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721322    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721647 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721616    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721694    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.721753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.721731    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\") pod \"2b49d3d4-fa91-4336-a255-d691720f477e\" (UID: \"2b49d3d4-fa91-4336-a255-d691720f477e\") "
Nov 02 08:31:08.729529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.722478    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b49d3d4-fa91-4336-a255-d691720f477e-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.732965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.732932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:08.733135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.733053    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "kube-api-access-gnqf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:08.733887 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.733215    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:08.734042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.733609    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:08.734160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.734120    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:08.734254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.734207    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:08.734526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.734501    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:08.735262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.735237    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo" pod "2b49d3d4-fa91-4336-a255-d691720f477e" (UID: "2b49d3d4-fa91-4336-a255-d691720f477e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:31:08.823851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823816    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.823851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823846    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.823851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823856    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqf9\" (UniqueName: \"kubernetes.io/projected/2b49d3d4-fa91-4336-a255-d691720f477e-kube-api-access-gnqf9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.824102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823867    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.824102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823875    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.824102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823884    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b49d3d4-fa91-4336-a255-d691720f477e-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.824102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823892    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:08.824102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:08.823899    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b49d3d4-fa91-4336-a255-d691720f477e-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:09.313802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.313766    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 08:31:09.347856 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.347821    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 08:31:09.378327 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.378282    4515 generic.go:356] "Generic (PLEG): container finished" podID="2b49d3d4-fa91-4336-a255-d691720f477e" containerID="cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d" exitCode=0
Nov 02 08:31:09.378497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.378374    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"
Nov 02 08:31:09.378497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.378373    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerDied","Data":"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"}
Nov 02 08:31:09.378594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.378502    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89" event={"ID":"2b49d3d4-fa91-4336-a255-d691720f477e","Type":"ContainerDied","Data":"61430da52b4e4fd1cd51f00efc6b86f3d4186623d91f2bf8751757cfc6f3a744"}
Nov 02 08:31:09.378594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.378535    4515 scope.go:122] "RemoveContainer" containerID="cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"
Nov 02 08:31:09.392054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.392036    4515 scope.go:122] "RemoveContainer" containerID="c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.409059    4515 scope.go:122] "RemoveContainer" containerID="cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:09.409684    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d\": container with ID starting with cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d not found: ID does not exist" containerID="cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.409757    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d"} err="failed to get container status \"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d\": rpc error: code = NotFound desc = could not find container \"cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d\": container with ID starting with cab8cd3d5a122980762871bb2fb0302b3d4fa22eb8f369223ced93a48d5ec94d not found: ID does not exist"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.409790    4515 scope.go:122] "RemoveContainer" containerID="c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:09.410207    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646\": container with ID starting with c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646 not found: ID does not exist" containerID="c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"
Nov 02 08:31:09.419559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.410278    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646"} err="failed to get container status \"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646\": rpc error: code = NotFound desc = could not find container \"c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646\": container with ID starting with c7d0f55776a1beec3a01c7bd56f79bd804a0fce9165cc2add5d8dc24431ec646 not found: ID does not exist"
Nov 02 08:31:09.434048 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.434013    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:31:09.443200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:09.443163    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2365fa-openshift-default-d56cdc89c-h2x89"]
Nov 02 08:31:11.001603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:11.001566    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:31:11.375503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:11.375407    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" path="/var/lib/kubelet/pods/2b49d3d4-fa91-4336-a255-d691720f477e/volumes"
Nov 02 08:31:11.654484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:11.654380    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:31:11.654754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:11.654677    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy" containerID="cri-o://927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee" gracePeriod=30
Nov 02 08:31:14.314536 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:14.314488    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing" containerID="cri-o://35193ee2ecc522070c9d54d1e51722c91d4545c7b1360e57f565d0c002d6b170" gracePeriod=25
Nov 02 08:31:14.348140 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:14.348080    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing" containerID="cri-o://55ff935c12c2dd648b4ccb03ac13db1e82bbd76a7e7f87b6559e37f5be02a8f5" gracePeriod=25
Nov 02 08:31:15.988363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.988325    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:15.991080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991046    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:31:15.991722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991704    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.991722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991722    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991732    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991738    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991745    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" containerName="maas-api"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991751    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" containerName="maas-api"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991760    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991765    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991771    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991775    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991782    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991787    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991796    4515 cpu_manager.go:414] "containerMap: removing container" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991800    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991806    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991811    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991820    4515 cpu_manager.go:414] "containerMap: removing container" podUID="98fba58e-785e-4418-a949-d99701bff54f" containerName="payload-pre-processing"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991825    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fba58e-785e-4418-a949-d99701bff54f" containerName="payload-pre-processing"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991832    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991837    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991847    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991852    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.991849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991858    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991862    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991869    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991873    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991881    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991886    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991893    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991897    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991910    4515 cpu_manager.go:414] "containerMap: removing container" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991914    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991921    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991926    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991937    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991944    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991954    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" containerName="payload-processing"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991960    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" containerName="payload-processing"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991973    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991977    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991983    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.991988    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992000    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992005    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992014    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992019    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992027    4515 cpu_manager.go:414] "containerMap: removing container" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992032    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992127    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992136    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992143    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992148    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992155    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992160    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992167    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992172    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992177    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992183    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992187    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992192    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3a587a-54db-4b3a-b2cb-62dbdcf91179" containerName="maas-api"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992200    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992205    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11c1b49-75dd-4ee4-ba46-af5a755cb572" containerName="payload-processing"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992212    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fba58e-785e-4418-a949-d99701bff54f" containerName="payload-pre-processing"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992217    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992222    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992228    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992233    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b922b513-40e7-4073-a7a2-aaf7d1dbb081" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992240    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992244    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992249    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:15.992579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.992254    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ce8faa-86cd-4d3a-bb6d-794cbb5d47c6" containerName="payload-processing"
Nov 02 08:31:15.997466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:15.997444    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.018127 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.018097    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:31:16.024739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.024705    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-3d8293-dockercfg-bmqhp\""
Nov 02 08:31:16.025064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.025048    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-3d8293\""
Nov 02 08:31:16.165564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.165531    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:31:16.166909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.166877    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:16.167101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167087    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b49d3d4-fa91-4336-a255-d691720f477e" containerName="istio-proxy"
Nov 02 08:31:16.167329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167293    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:16.167503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167485    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:16.167597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167586    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:16.167831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167675    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:16.167991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.167976    4515 cpu_manager.go:414] "containerMap: removing container" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:16.168085 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168074    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:16.168194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168182    4515 cpu_manager.go:414] "containerMap: removing container" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:16.168277 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168255    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:16.168373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168300    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:16.168373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168322    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:16.168515 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168501    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43443cca-aef0-479b-9fd2-c7545cdd676a" containerName="cleanup"
Nov 02 08:31:16.168577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168521    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78369e14-b18f-4ad8-a30b-32c1b99b49d7" containerName="cleanup"
Nov 02 08:31:16.168577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168536    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3596205-93a5-4e51-962b-9e49ccef4aa8" containerName="cleanup"
Nov 02 08:31:16.168577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168550    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f5a163-b184-4d6c-839c-9f239200f19a" containerName="cleanup"
Nov 02 08:31:16.168577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168564    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b45d301-a087-40d2-8b4e-4f49130cb648" containerName="cleanup"
Nov 02 08:31:16.168577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168574    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7ea61f-ef56-4699-a8df-a84332efb81e" containerName="cleanup"
Nov 02 08:31:16.168832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.168586    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff23e2c0-13a3-4364-b2db-62d7aa327278" containerName="cleanup"
Nov 02 08:31:16.176343 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.176241    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkv9\" (UniqueName: \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.176760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.176735    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.203448 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.198605    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:31:16.203639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.200518    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.278048 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.278009    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.278266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.278057    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5c6\" (UniqueName: \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.278354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.278326    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkv9\" (UniqueName: \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.278416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.278394    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.278784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.278765    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.301529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.301500    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkv9\" (UniqueName: \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\") pod \"payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.313580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.313544    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:16.384356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.380251    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.384356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.380362    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5c6\" (UniqueName: \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.384356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.381576    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.405043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.405006    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5c6\" (UniqueName: \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\") pod \"payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.520554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.520515    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:16.672960 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:31:16.672926    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b9c9c6_e90b_4b5c_ac17_3322df328f6a.slice/crio-031ff2e6cb8bc1c28837dbd20ecfb751acf23f889f7b1c8927609560b0748e74 WatchSource:0}: Error finding container 031ff2e6cb8bc1c28837dbd20ecfb751acf23f889f7b1c8927609560b0748e74: Status 404 returned error can't find the container with id 031ff2e6cb8bc1c28837dbd20ecfb751acf23f889f7b1c8927609560b0748e74
Nov 02 08:31:16.673590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.673514    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:31:16.834834 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:16.834806    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:31:16.869347 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:31:16.869280    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498e4915_9ad3_441d_95fb_21ea9534d23d.slice/crio-00f181efb1ef3121df288350bf751a3daea43e94c173483f4bc3898e31cf64a4 WatchSource:0}: Error finding container 00f181efb1ef3121df288350bf751a3daea43e94c173483f4bc3898e31cf64a4: Status 404 returned error can't find the container with id 00f181efb1ef3121df288350bf751a3daea43e94c173483f4bc3898e31cf64a4
Nov 02 08:31:17.051619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.051590    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:31:17.191810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191717    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.191810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191763    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191870    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191889    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191943    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.191974    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192073    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192142    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192213    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\") pod \"1485c7f4-d98c-4219-af21-62bbe779f6a9\" (UID: \"1485c7f4-d98c-4219-af21-62bbe779f6a9\") "
Nov 02 08:31:17.192274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192215    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:17.192274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192238    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:17.192603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192455    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:17.192603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192476    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:17.192708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192657    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:17.192982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192962    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.193070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.192987    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.193070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.193011    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.193070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.193025    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1485c7f4-d98c-4219-af21-62bbe779f6a9-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.193070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.193038    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.194395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.194368    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:31:17.194395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.194375    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:17.194643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.194626    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "kube-api-access-bg4xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:17.194685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.194654    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token" pod "1485c7f4-d98c-4219-af21-62bbe779f6a9" (UID: "1485c7f4-d98c-4219-af21-62bbe779f6a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:17.294045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.294008    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.294045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.294039    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.294045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.294048    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1485c7f4-d98c-4219-af21-62bbe779f6a9-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.294296 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.294058    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg4xk\" (UniqueName: \"kubernetes.io/projected/1485c7f4-d98c-4219-af21-62bbe779f6a9-kube-api-access-bg4xk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:17.438505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.438463    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" event={"ID":"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a","Type":"ContainerStarted","Data":"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"}
Nov 02 08:31:17.438715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.438532    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" event={"ID":"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a","Type":"ContainerStarted","Data":"031ff2e6cb8bc1c28837dbd20ecfb751acf23f889f7b1c8927609560b0748e74"}
Nov 02 08:31:17.438715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.438563    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:17.440361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.440327    4515 generic.go:356] "Generic (PLEG): container finished" podID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerID="927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee" exitCode=0
Nov 02 08:31:17.440495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.440379    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"
Nov 02 08:31:17.440495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.440417    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerDied","Data":"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"}
Nov 02 08:31:17.440495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.440466    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj" event={"ID":"1485c7f4-d98c-4219-af21-62bbe779f6a9","Type":"ContainerDied","Data":"8a8945a9378baa10c50ad58b7973544f3d729df7de2a87db07e29f0a9cd32473"}
Nov 02 08:31:17.440632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.440495    4515 scope.go:122] "RemoveContainer" containerID="927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"
Nov 02 08:31:17.442504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.442398    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" event={"ID":"498e4915-9ad3-441d-95fb-21ea9534d23d","Type":"ContainerStarted","Data":"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"}
Nov 02 08:31:17.442504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.442448    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" event={"ID":"498e4915-9ad3-441d-95fb-21ea9534d23d","Type":"ContainerStarted","Data":"00f181efb1ef3121df288350bf751a3daea43e94c173483f4bc3898e31cf64a4"}
Nov 02 08:31:17.442504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.442495    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:17.451703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.451686    4515 scope.go:122] "RemoveContainer" containerID="25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"
Nov 02 08:31:17.461641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.461618    4515 scope.go:122] "RemoveContainer" containerID="927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"
Nov 02 08:31:17.461935 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:17.461915    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee\": container with ID starting with 927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee not found: ID does not exist" containerID="927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"
Nov 02 08:31:17.462004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.461944    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee"} err="failed to get container status \"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee\": rpc error: code = NotFound desc = could not find container \"927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee\": container with ID starting with 927def18a812dff6872def1b576c8dca1a2f89394a29cd16f639b2c2388753ee not found: ID does not exist"
Nov 02 08:31:17.462004 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.461963    4515 scope.go:122] "RemoveContainer" containerID="25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"
Nov 02 08:31:17.462259 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:17.462240    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66\": container with ID starting with 25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66 not found: ID does not exist" containerID="25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"
Nov 02 08:31:17.462322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.462264    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66"} err="failed to get container status \"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66\": rpc error: code = NotFound desc = could not find container \"25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66\": container with ID starting with 25309dd9617f949e2cc10d9a6aeaa6a2a2c8e10ca4e10f3bc678c264f1112b66 not found: ID does not exist"
Nov 02 08:31:17.482046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.481990    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" podStartSLOduration=2.481975617 podStartE2EDuration="2.481975617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:17.479173315 +0000 UTC m=+4022.241610220" watchObservedRunningTime="2026-11-02 08:31:17.481975617 +0000 UTC m=+4022.244412523"
Nov 02 08:31:17.518975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.518933    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:31:17.530902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.530873    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4b85d50c-openshift-default-558d957877-z2knj"]
Nov 02 08:31:17.569541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:17.569473    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" podStartSLOduration=1.569456733 podStartE2EDuration="1.569456733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:17.568475478 +0000 UTC m=+4022.330912384" watchObservedRunningTime="2026-11-02 08:31:17.569456733 +0000 UTC m=+4022.331893681"
Nov 02 08:31:17.730472 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:17.730379    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:31:19.377242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:19.377207    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" path="/var/lib/kubelet/pods/1485c7f4-d98c-4219-af21-62bbe779f6a9/volumes"
Nov 02 08:31:22.887563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:22.887512    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:22.888001 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:22.887531    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:25.688894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.688826    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 08:31:25.712172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.712140    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\" (UniqueName: \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\") pod \"1fecf749-d5c7-44ab-8831-b79bf840d334\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") "
Nov 02 08:31:25.712357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.712201    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\") pod \"1fecf749-d5c7-44ab-8831-b79bf840d334\" (UID: \"1fecf749-d5c7-44ab-8831-b79bf840d334\") "
Nov 02 08:31:25.712680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.712655    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume" pod "1fecf749-d5c7-44ab-8831-b79bf840d334" (UID: "1fecf749-d5c7-44ab-8831-b79bf840d334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:25.712843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.712817    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fecf749-d5c7-44ab-8831-b79bf840d334-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:25.714354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.714331    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g" pod "1fecf749-d5c7-44ab-8831-b79bf840d334" (UID: "1fecf749-d5c7-44ab-8831-b79bf840d334"). InnerVolumeSpecName "kube-api-access-vsn9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:25.813757 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:25.813722    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsn9g\" (UniqueName: \"kubernetes.io/projected/1fecf749-d5c7-44ab-8831-b79bf840d334-kube-api-access-vsn9g\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:26.503393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.503351    4515 generic.go:356] "Generic (PLEG): container finished" podID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerID="baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b" exitCode=0
Nov 02 08:31:26.503609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.503414    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"
Nov 02 08:31:26.503609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.503447    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" event={"ID":"1fecf749-d5c7-44ab-8831-b79bf840d334","Type":"ContainerDied","Data":"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"}
Nov 02 08:31:26.503609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.503499    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8" event={"ID":"1fecf749-d5c7-44ab-8831-b79bf840d334","Type":"ContainerDied","Data":"b737f94114b654f4d7a223662bec6eb19ff35c573d8cd45fd70883976fc1f305"}
Nov 02 08:31:26.503609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.503522    4515 scope.go:122] "RemoveContainer" containerID="baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"
Nov 02 08:31:26.514507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.514490    4515 scope.go:122] "RemoveContainer" containerID="baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"
Nov 02 08:31:26.514799 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:26.514779    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b\": container with ID starting with baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b not found: ID does not exist" containerID="baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"
Nov 02 08:31:26.514867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.514814    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b"} err="failed to get container status \"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b\": rpc error: code = NotFound desc = could not find container \"baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b\": container with ID starting with baae3eb7a4eb3932eadc13b63a816584d61f00f41c795ee7b57649a6e3fa1e9b not found: ID does not exist"
Nov 02 08:31:26.556286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.556251    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 08:31:26.559231 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:26.559208    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3575b9-7d86dfd575-rdtf8"]
Nov 02 08:31:27.373639 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:27.373604    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" path="/var/lib/kubelet/pods/1fecf749-d5c7-44ab-8831-b79bf840d334/volumes"
Nov 02 08:31:28.452858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:28.452830    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:31:28.453363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:28.453095    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:31:32.887973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:32.887932    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:32.889103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:32.887945    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:34.565747 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.565703    4515 generic.go:356] "Generic (PLEG): container finished" podID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerID="35193ee2ecc522070c9d54d1e51722c91d4545c7b1360e57f565d0c002d6b170" exitCode=0
Nov 02 08:31:34.566255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.565775    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" event={"ID":"63d2b70c-a401-4493-bac9-fd92f603ca7b","Type":"ContainerDied","Data":"35193ee2ecc522070c9d54d1e51722c91d4545c7b1360e57f565d0c002d6b170"}
Nov 02 08:31:34.567683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.567657    4515 generic.go:356] "Generic (PLEG): container finished" podID="0cd1a400-35fd-439a-9f00-105658388f10" containerID="55ff935c12c2dd648b4ccb03ac13db1e82bbd76a7e7f87b6559e37f5be02a8f5" exitCode=0
Nov 02 08:31:34.567808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.567730    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" event={"ID":"0cd1a400-35fd-439a-9f00-105658388f10","Type":"ContainerDied","Data":"55ff935c12c2dd648b4ccb03ac13db1e82bbd76a7e7f87b6559e37f5be02a8f5"}
Nov 02 08:31:34.659428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.658534    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 08:31:34.696208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.696181    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 08:31:34.764893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.764853    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\" (UniqueName: \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\") pod \"63d2b70c-a401-4493-bac9-fd92f603ca7b\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") "
Nov 02 08:31:34.765096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765000    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\") pod \"63d2b70c-a401-4493-bac9-fd92f603ca7b\" (UID: \"63d2b70c-a401-4493-bac9-fd92f603ca7b\") "
Nov 02 08:31:34.765191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765134    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\" (UniqueName: \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\") pod \"0cd1a400-35fd-439a-9f00-105658388f10\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") "
Nov 02 08:31:34.765254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765218    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\") pod \"0cd1a400-35fd-439a-9f00-105658388f10\" (UID: \"0cd1a400-35fd-439a-9f00-105658388f10\") "
Nov 02 08:31:34.765556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765502    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume" pod "63d2b70c-a401-4493-bac9-fd92f603ca7b" (UID: "63d2b70c-a401-4493-bac9-fd92f603ca7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:34.765716 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765618    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume" pod "0cd1a400-35fd-439a-9f00-105658388f10" (UID: "0cd1a400-35fd-439a-9f00-105658388f10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:34.765928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765910    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d2b70c-a401-4493-bac9-fd92f603ca7b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:34.765997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.765930    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd1a400-35fd-439a-9f00-105658388f10-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:34.767762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.767740    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c" pod "63d2b70c-a401-4493-bac9-fd92f603ca7b" (UID: "63d2b70c-a401-4493-bac9-fd92f603ca7b"). InnerVolumeSpecName "kube-api-access-vw72c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:34.767877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.767757    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh" pod "0cd1a400-35fd-439a-9f00-105658388f10" (UID: "0cd1a400-35fd-439a-9f00-105658388f10"). InnerVolumeSpecName "kube-api-access-vd2nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:34.867900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.867819    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw72c\" (UniqueName: \"kubernetes.io/projected/63d2b70c-a401-4493-bac9-fd92f603ca7b-kube-api-access-vw72c\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:34.867900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:34.867858    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd2nh\" (UniqueName: \"kubernetes.io/projected/0cd1a400-35fd-439a-9f00-105658388f10-kube-api-access-vd2nh\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:35.576283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.576251    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"
Nov 02 08:31:35.576283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.576271    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f" event={"ID":"63d2b70c-a401-4493-bac9-fd92f603ca7b","Type":"ContainerDied","Data":"e172585907ca640f2e26d520347267d1ca6b15851be15ce9cd96ee1406afa6da"}
Nov 02 08:31:35.576817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.576360    4515 scope.go:122] "RemoveContainer" containerID="35193ee2ecc522070c9d54d1e51722c91d4545c7b1360e57f565d0c002d6b170"
Nov 02 08:31:35.578266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.578235    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"
Nov 02 08:31:35.578424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.578235    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5" event={"ID":"0cd1a400-35fd-439a-9f00-105658388f10","Type":"ContainerDied","Data":"4f243b001ae8827c7acc8219eb87c8261f2251e6e653e2cc8c2587b43607b6ca"}
Nov 02 08:31:35.587945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.587926    4515 scope.go:122] "RemoveContainer" containerID="55ff935c12c2dd648b4ccb03ac13db1e82bbd76a7e7f87b6559e37f5be02a8f5"
Nov 02 08:31:35.629043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.629001    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 08:31:35.645111 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.645067    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e4c131-57f569c5cc-7q57f"]
Nov 02 08:31:35.667680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.667645    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 08:31:35.670466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:35.670441    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e4c131-5c4fdcdc59-x24p5"]
Nov 02 08:31:37.374725 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:37.374691    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd1a400-35fd-439a-9f00-105658388f10" path="/var/lib/kubelet/pods/0cd1a400-35fd-439a-9f00-105658388f10/volumes"
Nov 02 08:31:37.375163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:37.375149    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" path="/var/lib/kubelet/pods/63d2b70c-a401-4493-bac9-fd92f603ca7b/volumes"
Nov 02 08:31:39.349255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.349219    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 08:31:39.901990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.901952    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:31:39.902934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.902910    4515 cpu_manager.go:414] "containerMap: removing container" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing"
Nov 02 08:31:39.903045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.902939    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing"
Nov 02 08:31:39.903045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.902974    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing"
Nov 02 08:31:39.903045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.902983    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing"
Nov 02 08:31:39.903045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903036    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing"
Nov 02 08:31:39.903045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903044    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing"
Nov 02 08:31:39.903235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903067    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903075    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903092    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903101    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903243    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1a400-35fd-439a-9f00-105658388f10" containerName="payload-pre-processing"
Nov 02 08:31:39.903426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903258    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903269    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1485c7f4-d98c-4219-af21-62bbe779f6a9" containerName="istio-proxy"
Nov 02 08:31:39.903426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903278    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d2b70c-a401-4493-bac9-fd92f603ca7b" containerName="payload-processing"
Nov 02 08:31:39.903426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.903291    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fecf749-d5c7-44ab-8831-b79bf840d334" containerName="payload-pre-processing"
Nov 02 08:31:39.908605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.908582    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:39.930002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.929975    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"rate-test-model-56bbb6-kserve-self-signed-certs\""
Nov 02 08:31:39.930256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.930074    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"default-dockercfg-j4wmw\""
Nov 02 08:31:39.932151 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.932131    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:31:39.939226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.939192    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"openshift-service-ca.crt\""
Nov 02 08:31:39.939385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:39.939296    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56bbb6\"/\"kube-root-ca.crt\""
Nov 02 08:31:40.039130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039091    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.039379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039203    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.039379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039270    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.039379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039376    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.039595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039546    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.039652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.039613    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klqb9\" (UniqueName: \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.144940    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145052    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145154    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145355    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145456    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145557    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.145630    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klqb9\" (UniqueName: \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.146262    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.147354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.146637    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.157406 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.148999    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.157406 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.150421    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.171871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.171832    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klqb9\" (UniqueName: \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\") pod \"rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") " pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.231287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.231247    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:40.538058 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:31:40.538028    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d6b8eda_0315_4668_abed_16075315bd17.slice/crio-6dc6102c60ef9d0d46500ff9bda55e4ba76cfb019ee0acec332440485e954d66 WatchSource:0}: Error finding container 6dc6102c60ef9d0d46500ff9bda55e4ba76cfb019ee0acec332440485e954d66: Status 404 returned error can't find the container with id 6dc6102c60ef9d0d46500ff9bda55e4ba76cfb019ee0acec332440485e954d66
Nov 02 08:31:40.539068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.539044    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:31:40.619117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:40.619088    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" event={"ID":"0d6b8eda-0315-4668-abed-16075315bd17","Type":"ContainerStarted","Data":"6dc6102c60ef9d0d46500ff9bda55e4ba76cfb019ee0acec332440485e954d66"}
Nov 02 08:31:41.626778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:41.626739    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" event={"ID":"0d6b8eda-0315-4668-abed-16075315bd17","Type":"ContainerStarted","Data":"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"}
Nov 02 08:31:41.669710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:41.669660    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" podStartSLOduration=2.669645868 podStartE2EDuration="2.669645868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:41.666756376 +0000 UTC m=+4046.429193287" watchObservedRunningTime="2026-11-02 08:31:41.669645868 +0000 UTC m=+4046.432082773"
Nov 02 08:31:42.231493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:42.231459    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:46.540848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:46.540796    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 08:31:46.541411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:46.541041    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" containerID="cri-o://b99fd3f0dca6d612d2b4a5b860cb583220999c7413b1ced03e0d676a158ef3ce" gracePeriod=30
Nov 02 08:31:47.730492 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:47.730447    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:31:49.289047 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:49.289009    4515 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:31:49.289490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:49.289072    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:31:50.231924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:50.231885    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:50.244932 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:50.244907    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:50.702534 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:50.702503    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:31:51.702709 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.702673    4515 generic.go:356] "Generic (PLEG): container finished" podID="03afe710-f612-4504-9c63-e282b67993f8" containerID="b99fd3f0dca6d612d2b4a5b860cb583220999c7413b1ced03e0d676a158ef3ce" exitCode=0
Nov 02 08:31:51.703210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.702755    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerDied","Data":"b99fd3f0dca6d612d2b4a5b860cb583220999c7413b1ced03e0d676a158ef3ce"}
Nov 02 08:31:51.703210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.702840    4515 scope.go:122] "RemoveContainer" containerID="1efd6b2fdf359b1dc9f1372b7f1d87a67d3c80e65bb2a9a7610bd1b4c4802d7c"
Nov 02 08:31:51.828961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.828934    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 08:31:51.913003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.912971    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913037    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913070    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913124    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913150    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913189    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913266    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913339    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:51.913451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913401    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913631    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:51.913685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913651    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\") pod \"03afe710-f612-4504-9c63-e282b67993f8\" (UID: \"03afe710-f612-4504-9c63-e282b67993f8\") "
Nov 02 08:31:51.913807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913677    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:51.913807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913741    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:51.914261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.913976    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:51.914407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.914288    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03afe710-f612-4504-9c63-e282b67993f8-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:51.914407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.914345    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:51.914407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.914363    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:51.914407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.914377    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:51.914407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.914391    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:51.915721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.915697    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:51.916070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.916049    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:31:51.916374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.916353    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:51.916501 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:51.916477    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz" pod "03afe710-f612-4504-9c63-e282b67993f8" (UID: "03afe710-f612-4504-9c63-e282b67993f8"). InnerVolumeSpecName "kube-api-access-zqptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:52.015339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.015275    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:52.015339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.015322    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03afe710-f612-4504-9c63-e282b67993f8-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:52.015339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.015332    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03afe710-f612-4504-9c63-e282b67993f8-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:52.015339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.015342    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqptz\" (UniqueName: \"kubernetes.io/projected/03afe710-f612-4504-9c63-e282b67993f8-kube-api-access-zqptz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:52.710897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.710857    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5" event={"ID":"03afe710-f612-4504-9c63-e282b67993f8","Type":"ContainerDied","Data":"38b96ec18173186a92c6668667d4936b894b8d65db4e0b0a30e72ca2c342e242"}
Nov 02 08:31:52.711410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.710917    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"
Nov 02 08:31:52.711410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.710922    4515 scope.go:122] "RemoveContainer" containerID="b99fd3f0dca6d612d2b4a5b860cb583220999c7413b1ced03e0d676a158ef3ce"
Nov 02 08:31:52.775793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.775759    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 08:31:52.788128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:52.788091    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e4c131-openshift-default-6bfc86fbcf-k8rq5"]
Nov 02 08:31:53.374458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:53.374423    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03afe710-f612-4504-9c63-e282b67993f8" path="/var/lib/kubelet/pods/03afe710-f612-4504-9c63-e282b67993f8/volumes"
Nov 02 08:31:53.696707 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:53.696618    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 08:31:53.737927 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:53.737891    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 08:31:54.375070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.375016    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main" containerID="cri-o://49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581" gracePeriod=45
Nov 02 08:31:54.645063 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.644995    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 08:31:54.729294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.729255    4515 generic.go:356] "Generic (PLEG): container finished" podID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerID="49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581" exitCode=0
Nov 02 08:31:54.729580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.729297    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"
Nov 02 08:31:54.729580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.729363    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" event={"ID":"307abf08-13a0-410e-bd8e-39a7c69d707f","Type":"ContainerDied","Data":"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"}
Nov 02 08:31:54.729580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.729409    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52" event={"ID":"307abf08-13a0-410e-bd8e-39a7c69d707f","Type":"ContainerDied","Data":"427ede97cb06fbe40a3834028aa8e47257a5e0c37d8d427a1723805b881d59b0"}
Nov 02 08:31:54.729580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.729433    4515 scope.go:122] "RemoveContainer" containerID="49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"
Nov 02 08:31:54.739461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.739436    4515 scope.go:122] "RemoveContainer" containerID="49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"
Nov 02 08:31:54.739787 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:54.739768    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581\": container with ID starting with 49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581 not found: ID does not exist" containerID="49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"
Nov 02 08:31:54.739829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.739806    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581"} err="failed to get container status \"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581\": rpc error: code = NotFound desc = could not find container \"49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581\": container with ID starting with 49c39d027864d4a5d412aa80d95bb3d0be9d57f5f0dc3fdab8ef379adac62581 not found: ID does not exist"
Nov 02 08:31:54.755181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755156    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755200    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\" (UniqueName: \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755251    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755304    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755386    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755552 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755449    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\" (UniqueName: \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\") pod \"307abf08-13a0-410e-bd8e-39a7c69d707f\" (UID: \"307abf08-13a0-410e-bd8e-39a7c69d707f\") "
Nov 02 08:31:54.755619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755594    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.755685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755617    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.755776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.755747    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.756706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.756678    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.756706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.756708    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.756895 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.756723    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.757914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.757667    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.757914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.757852    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "kube-api-access-hfz56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:54.758127 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.758103    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs" pod "307abf08-13a0-410e-bd8e-39a7c69d707f" (UID: "307abf08-13a0-410e-bd8e-39a7c69d707f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:54.858041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.857986    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307abf08-13a0-410e-bd8e-39a7c69d707f-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.858254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.858061    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfz56\" (UniqueName: \"kubernetes.io/projected/307abf08-13a0-410e-bd8e-39a7c69d707f-kube-api-access-hfz56\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.858254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:54.858078    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307abf08-13a0-410e-bd8e-39a7c69d707f-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.079208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:55.079175    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 08:31:55.092633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:55.092603    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-2-w0-0fbc17-kserve-5bb5958775-9kn52"]
Nov 02 08:31:55.374161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:55.374082    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" path="/var/lib/kubelet/pods/307abf08-13a0-410e-bd8e-39a7c69d707f/volumes"
Nov 02 08:31:56.386409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:56.386366    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:31:58.697405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:58.697363    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" podUID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" containerName="payload-processing" containerID="cri-o://e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6" gracePeriod=25
Nov 02 08:31:58.738453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:58.738400    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" podUID="492d1166-1eb2-4523-a33f-80045ef3b210" containerName="payload-pre-processing" containerID="cri-o://7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151" gracePeriod=25
Nov 02 08:31:59.032496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.032464    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 08:31:59.060592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.060568    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 08:31:59.126290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.126256    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\" (UniqueName: \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\") pod \"492d1166-1eb2-4523-a33f-80045ef3b210\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") "
Nov 02 08:31:59.126493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.126352    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\" (UniqueName: \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\") pod \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") "
Nov 02 08:31:59.126493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.126415    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\") pod \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\" (UID: \"5402fa8e-57da-45a6-a603-bb2da47ba2d6\") "
Nov 02 08:31:59.126584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.126489    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\") pod \"492d1166-1eb2-4523-a33f-80045ef3b210\" (UID: \"492d1166-1eb2-4523-a33f-80045ef3b210\") "
Nov 02 08:31:59.127092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.127058    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume" pod "5402fa8e-57da-45a6-a603-bb2da47ba2d6" (UID: "5402fa8e-57da-45a6-a603-bb2da47ba2d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:59.127259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.127138    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5402fa8e-57da-45a6-a603-bb2da47ba2d6-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.127514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.127488    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume" pod "492d1166-1eb2-4523-a33f-80045ef3b210" (UID: "492d1166-1eb2-4523-a33f-80045ef3b210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:59.129071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.129043    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6" pod "5402fa8e-57da-45a6-a603-bb2da47ba2d6" (UID: "5402fa8e-57da-45a6-a603-bb2da47ba2d6"). InnerVolumeSpecName "kube-api-access-pdrv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:59.129163 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.129087    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42" pod "492d1166-1eb2-4523-a33f-80045ef3b210" (UID: "492d1166-1eb2-4523-a33f-80045ef3b210"). InnerVolumeSpecName "kube-api-access-x7x42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:59.228567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.228533    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492d1166-1eb2-4523-a33f-80045ef3b210-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.228567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.228561    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7x42\" (UniqueName: \"kubernetes.io/projected/492d1166-1eb2-4523-a33f-80045ef3b210-kube-api-access-x7x42\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.228567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.228571    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdrv6\" (UniqueName: \"kubernetes.io/projected/5402fa8e-57da-45a6-a603-bb2da47ba2d6-kube-api-access-pdrv6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.765201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.765165    4515 generic.go:356] "Generic (PLEG): container finished" podID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" containerID="e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6" exitCode=0
Nov 02 08:31:59.765738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.765218    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"
Nov 02 08:31:59.765738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.765258    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" event={"ID":"5402fa8e-57da-45a6-a603-bb2da47ba2d6","Type":"ContainerDied","Data":"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"}
Nov 02 08:31:59.765738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.765327    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl" event={"ID":"5402fa8e-57da-45a6-a603-bb2da47ba2d6","Type":"ContainerDied","Data":"8c0ee934b05f2d68d21c489519c94911f196d80d0ecadf8902674233f14d8fc4"}
Nov 02 08:31:59.765738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.765358    4515 scope.go:122] "RemoveContainer" containerID="e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"
Nov 02 08:31:59.767049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.767022    4515 generic.go:356] "Generic (PLEG): container finished" podID="492d1166-1eb2-4523-a33f-80045ef3b210" containerID="7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151" exitCode=0
Nov 02 08:31:59.767177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.767080    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"
Nov 02 08:31:59.767177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.767128    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" event={"ID":"492d1166-1eb2-4523-a33f-80045ef3b210","Type":"ContainerDied","Data":"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"}
Nov 02 08:31:59.767293 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.767190    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t" event={"ID":"492d1166-1eb2-4523-a33f-80045ef3b210","Type":"ContainerDied","Data":"7c4ed5a5fc60610725f8b2084011dd5578e147237863dc64ac8b2b00d617a579"}
Nov 02 08:31:59.775624 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.775603    4515 scope.go:122] "RemoveContainer" containerID="e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"
Nov 02 08:31:59.775880 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:59.775862    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6\": container with ID starting with e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6 not found: ID does not exist" containerID="e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"
Nov 02 08:31:59.775937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.775888    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6"} err="failed to get container status \"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6\": rpc error: code = NotFound desc = could not find container \"e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6\": container with ID starting with e1dfec766f143b2bf857cd5546b79a875814f6c4933786af9c4e0db397dbcef6 not found: ID does not exist"
Nov 02 08:31:59.775937 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.775908    4515 scope.go:122] "RemoveContainer" containerID="7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"
Nov 02 08:31:59.784891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.784875    4515 scope.go:122] "RemoveContainer" containerID="7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"
Nov 02 08:31:59.785159 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:31:59.785138    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151\": container with ID starting with 7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151 not found: ID does not exist" containerID="7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"
Nov 02 08:31:59.785220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.785167    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151"} err="failed to get container status \"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151\": rpc error: code = NotFound desc = could not find container \"7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151\": container with ID starting with 7a6563babaab8da72d6c6514004a2d05378c56132f5e930b5f618022e59da151 not found: ID does not exist"
Nov 02 08:31:59.828462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.828427    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 08:31:59.840612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.840582    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-6929a4-6fcb8c55bc-cz97t"]
Nov 02 08:31:59.868553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.868521    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 08:31:59.872727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:31:59.872698    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-6929a4-6f857bbb47-mh7pl"]
Nov 02 08:32:01.374424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:01.374390    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492d1166-1eb2-4523-a33f-80045ef3b210" path="/var/lib/kubelet/pods/492d1166-1eb2-4523-a33f-80045ef3b210/volumes"
Nov 02 08:32:01.374848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:01.374833    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" path="/var/lib/kubelet/pods/5402fa8e-57da-45a6-a603-bb2da47ba2d6/volumes"
Nov 02 08:32:01.694764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:01.694672    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 08:32:01.694948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:01.694903    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" containerID="cri-o://a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087" gracePeriod=30
Nov 02 08:32:03.482494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:03.482459    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:32:03.482921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:03.482530    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:32:06.973882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:06.973816    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 08:32:07.034041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034005    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034097    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034123    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034158    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034296    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034372    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034399    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034394    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:07.034461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034428    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034606    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\") pod \"aba27247-c0eb-427b-9a7b-b765a793d789\" (UID: \"aba27247-c0eb-427b-9a7b-b765a793d789\") "
Nov 02 08:32:07.034760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.034711    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:07.035138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.035102    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:07.035286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.035166    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.035286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.035186    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.035567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.035424    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:07.035811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.035783    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:07.037206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.037181    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:32:07.037530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.037496    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:07.037889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.037865    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:07.038224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.038202    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn" pod "aba27247-c0eb-427b-9a7b-b765a793d789" (UID: "aba27247-c0eb-427b-9a7b-b765a793d789"). InnerVolumeSpecName "kube-api-access-dv4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:07.123111 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.123075    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136374    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136412    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba27247-c0eb-427b-9a7b-b765a793d789-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136426    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba27247-c0eb-427b-9a7b-b765a793d789-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136441    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136456    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba27247-c0eb-427b-9a7b-b765a793d789-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136469    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4sn\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-kube-api-access-dv4sn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.136944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.136482    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba27247-c0eb-427b-9a7b-b765a793d789-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:07.141241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.141207    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 08:32:07.149951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.149922    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 08:32:07.821246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.821214    4515 generic.go:356] "Generic (PLEG): container finished" podID="aba27247-c0eb-427b-9a7b-b765a793d789" containerID="a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087" exitCode=0
Nov 02 08:32:07.821470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.821264    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"
Nov 02 08:32:07.821470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.821319    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" event={"ID":"aba27247-c0eb-427b-9a7b-b765a793d789","Type":"ContainerDied","Data":"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"}
Nov 02 08:32:07.821470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.821374    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc" event={"ID":"aba27247-c0eb-427b-9a7b-b765a793d789","Type":"ContainerDied","Data":"ee975c54551a55bab85d3e2ff1ef3aea58cf777b30f11b4dcdfa6f15bd74987f"}
Nov 02 08:32:07.821470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.821400    4515 scope.go:122] "RemoveContainer" containerID="a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"
Nov 02 08:32:07.832586 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.832565    4515 scope.go:122] "RemoveContainer" containerID="a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"
Nov 02 08:32:07.832869 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:07.832844    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087\": container with ID starting with a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087 not found: ID does not exist" containerID="a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"
Nov 02 08:32:07.832958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.832878    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087"} err="failed to get container status \"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087\": rpc error: code = NotFound desc = could not find container \"a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087\": container with ID starting with a7cbbfa0b13bbb423f3cedd7c023dda701efc1c2fc5018f916698bc77e713087 not found: ID does not exist"
Nov 02 08:32:07.886815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.886780    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 08:32:07.898978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:07.898949    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-98zdc"]
Nov 02 08:32:09.373730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:09.373697    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" path="/var/lib/kubelet/pods/aba27247-c0eb-427b-9a7b-b765a793d789/volumes"
Nov 02 08:32:17.723674 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:17.723638    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:32:22.148840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.148770    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main" containerID="cri-o://9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd" gracePeriod=45
Nov 02 08:32:22.163438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.163405    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" containerID="cri-o://ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c" gracePeriod=45
Nov 02 08:32:22.172727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.172697    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main" containerID="cri-o://798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47" gracePeriod=45
Nov 02 08:32:22.173123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.173100    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused"
Nov 02 08:32:22.491455 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.491421    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 08:32:22.548561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.548490    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 08:32:22.562711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.562681    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.577829    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.577876    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.577993    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.578039    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\" (UniqueName: \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.578084    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.578125    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\") pod \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\" (UID: \"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c\") "
Nov 02 08:32:22.578888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.578646    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.579925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.579873    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.580831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.580784    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.581655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.581621    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:22.582433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.582402    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.582661 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.582624    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69" pod "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" (UID: "6de626b4-2ae0-4ebc-95d8-9eb7ea71605c"). InnerVolumeSpecName "kube-api-access-5mz69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:22.679043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679006    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679053    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679126    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\" (UniqueName: \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679155    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679188    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\" (UniqueName: \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679258    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679295    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679406    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679422    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679470    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679528    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679555    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\") pod \"51f55e14-e075-4ea4-91f4-e53800132d1b\" (UID: \"51f55e14-e075-4ea4-91f4-e53800132d1b\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679582    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\") pod \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\" (UID: \"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56\") "
Nov 02 08:32:22.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679643    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.680100 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.679987    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.680100 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.680012    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.680350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.680287    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.680774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.680743    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.681252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681228    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681259    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mz69\" (UniqueName: \"kubernetes.io/projected/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-kube-api-access-5mz69\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681276    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681290    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681301    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681329    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681343    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681356    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681369    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681383    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681396    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.681977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.681410    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.682236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.682210    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "kube-api-access-bbpzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:22.682549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.682522    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "kube-api-access-lh4qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:22.683590 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.683565    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.683818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.683793    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:22.683867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.683800    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs" pod "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" (UID: "dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:22.683867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.683819    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs" pod "51f55e14-e075-4ea4-91f4-e53800132d1b" (UID: "51f55e14-e075-4ea4-91f4-e53800132d1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:22.782959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782927    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4qj\" (UniqueName: \"kubernetes.io/projected/51f55e14-e075-4ea4-91f4-e53800132d1b-kube-api-access-lh4qj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.782959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782957    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f55e14-e075-4ea4-91f4-e53800132d1b-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.783189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782967    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpzb\" (UniqueName: \"kubernetes.io/projected/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-kube-api-access-bbpzb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.783189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782977    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f55e14-e075-4ea4-91f4-e53800132d1b-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.783189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782987    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.783189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.782996    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:32:22.919278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.919240    4515 generic.go:356] "Generic (PLEG): container finished" podID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerID="9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd" exitCode=0
Nov 02 08:32:22.919511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.919293    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"
Nov 02 08:32:22.919511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.919286    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" event={"ID":"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c","Type":"ContainerDied","Data":"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"}
Nov 02 08:32:22.919511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.919369    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld" event={"ID":"6de626b4-2ae0-4ebc-95d8-9eb7ea71605c","Type":"ContainerDied","Data":"f5376b6af1a50450c5a79aa4ca1ac5ece63211e9081a15a694f24da995796a8d"}
Nov 02 08:32:22.919511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.919396    4515 scope.go:122] "RemoveContainer" containerID="9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"
Nov 02 08:32:22.921042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.921017    4515 generic.go:356] "Generic (PLEG): container finished" podID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerID="ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c" exitCode=0
Nov 02 08:32:22.921170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.921065    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"
Nov 02 08:32:22.921170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.921101    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" event={"ID":"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56","Type":"ContainerDied","Data":"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"}
Nov 02 08:32:22.921170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.921151    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh" event={"ID":"dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56","Type":"ContainerDied","Data":"8b91ad27475c0fca18fc9e6cf77ebab7a13c322fd0239676806147c1aa8f38f5"}
Nov 02 08:32:22.922704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.922669    4515 generic.go:356] "Generic (PLEG): container finished" podID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerID="798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47" exitCode=0
Nov 02 08:32:22.922821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.922737    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" event={"ID":"51f55e14-e075-4ea4-91f4-e53800132d1b","Type":"ContainerDied","Data":"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"}
Nov 02 08:32:22.922821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.922764    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4" event={"ID":"51f55e14-e075-4ea4-91f4-e53800132d1b","Type":"ContainerDied","Data":"52da50223492e5a832e2bbcbf3b7db71eeb74d62f67db447170cf84553896c99"}
Nov 02 08:32:22.922821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.922812    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"
Nov 02 08:32:22.933874 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.933856    4515 scope.go:122] "RemoveContainer" containerID="9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"
Nov 02 08:32:22.934254 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:22.934226    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd\": container with ID starting with 9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd not found: ID does not exist" containerID="9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"
Nov 02 08:32:22.934367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.934263    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd"} err="failed to get container status \"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd\": rpc error: code = NotFound desc = could not find container \"9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd\": container with ID starting with 9fba38d68a46bf461c52e2868e64511068f0109ee78f21d1e7135cfe5efe20fd not found: ID does not exist"
Nov 02 08:32:22.934367 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.934287    4515 scope.go:122] "RemoveContainer" containerID="ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"
Nov 02 08:32:22.952824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.952800    4515 scope.go:122] "RemoveContainer" containerID="ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"
Nov 02 08:32:22.953139 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:22.953110    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c\": container with ID starting with ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c not found: ID does not exist" containerID="ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"
Nov 02 08:32:22.953191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.953149    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c"} err="failed to get container status \"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c\": rpc error: code = NotFound desc = could not find container \"ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c\": container with ID starting with ef462f697e141243119e00ff19f0110425b944e5d848b11271e89a78e344492c not found: ID does not exist"
Nov 02 08:32:22.953191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.953171    4515 scope.go:122] "RemoveContainer" containerID="798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"
Nov 02 08:32:22.962490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.962458    4515 scope.go:122] "RemoveContainer" containerID="798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"
Nov 02 08:32:22.962901 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:22.962860    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47\": container with ID starting with 798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47 not found: ID does not exist" containerID="798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"
Nov 02 08:32:22.962975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:22.962928    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47"} err="failed to get container status \"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47\": rpc error: code = NotFound desc = could not find container \"798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47\": container with ID starting with 798872803ab7a88674b4ee1a2043662cc25b8573df0a122340c9cd648f845b47 not found: ID does not exist"
Nov 02 08:32:23.024707 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.024670    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 08:32:23.028202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.028177    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-2-w1-7e7a83-kserve-7f4978dccd-c5wld"]
Nov 02 08:32:23.068695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.068614    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 08:32:23.079137 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.079104    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/premium-w1-7e7a83-kserve-95764d486-bbtjh"]
Nov 02 08:32:23.115025 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.114987    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 08:32:23.118588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.118565    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/simulator-w1-7e7a83-kserve-966bfbb4c-bdst4"]
Nov 02 08:32:23.373151 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.373063    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" path="/var/lib/kubelet/pods/51f55e14-e075-4ea4-91f4-e53800132d1b/volumes"
Nov 02 08:32:23.373615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.373598    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" path="/var/lib/kubelet/pods/6de626b4-2ae0-4ebc-95d8-9eb7ea71605c/volumes"
Nov 02 08:32:23.373996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:23.373983    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" path="/var/lib/kubelet/pods/dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56/volumes"
Nov 02 08:32:24.961050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961015    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:32:24.961860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961837    4515 cpu_manager.go:414] "containerMap: removing container" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main"
Nov 02 08:32:24.961860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961861    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961878    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961887    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961940    4515 cpu_manager.go:414] "containerMap: removing container" podUID="492d1166-1eb2-4523-a33f-80045ef3b210" containerName="payload-pre-processing"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961949    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492d1166-1eb2-4523-a33f-80045ef3b210" containerName="payload-pre-processing"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961986    4515 cpu_manager.go:414] "containerMap: removing container" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.961994    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962021    4515 cpu_manager.go:414] "containerMap: removing container" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main"
Nov 02 08:32:24.962027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962029    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962040    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" containerName="payload-processing"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962048    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" containerName="payload-processing"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962062    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962069    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962084    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962091    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962213    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5402fa8e-57da-45a6-a603-bb2da47ba2d6" containerName="payload-processing"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962227    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de626b4-2ae0-4ebc-95d8-9eb7ea71605c" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962238    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962247    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc07ee5-9e04-4cbc-8d34-e4a898d4ba56" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962258    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="307abf08-13a0-410e-bd8e-39a7c69d707f" containerName="main"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962267    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962280    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="492d1166-1eb2-4523-a33f-80045ef3b210" containerName="payload-pre-processing"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962292    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba27247-c0eb-427b-9a7b-b765a793d789" containerName="istio-proxy"
Nov 02 08:32:24.962461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.962302    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f55e14-e075-4ea4-91f4-e53800132d1b" containerName="main"
Nov 02 08:32:24.966237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.966195    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:24.988935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:24.988905    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a44352-openshift-default-dockercfg-mg66s\""
Nov 02 08:32:25.005258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.005217    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:32:25.115704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115670    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.115904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115710    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.115904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115847    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.115904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115885    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7jn\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.116029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115946    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.116029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.115986    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.116115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.116057    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.116155 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.116115    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.116196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.116156    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.217922    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.217975    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218063    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218096    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218165    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218202    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7jn\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218248    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218287    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218379    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218482    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.218831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.218796    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.219518 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.219491    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.219665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.219578    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.219813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.219788    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.223933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.223910    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.228099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.228027    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.261871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.261838    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.262024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.261935    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7jn\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\") pod \"e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") " pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.280807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.280777    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:25.604815 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:32:25.604781    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9b6f41_306e_42e9_bcf8_9fc9985507cb.slice/crio-69b38a07751211885993f5b96d98cabf0d53437fa8b7424f79aaf0dc9fdd7fc5 WatchSource:0}: Error finding container 69b38a07751211885993f5b96d98cabf0d53437fa8b7424f79aaf0dc9fdd7fc5: Status 404 returned error can't find the container with id 69b38a07751211885993f5b96d98cabf0d53437fa8b7424f79aaf0dc9fdd7fc5
Nov 02 08:32:25.606870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.606845    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:32:25.607458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.607425    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:25.607535 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.607488    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:25.607535 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.607517    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:25.950902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.950796    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerStarted","Data":"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"}
Nov 02 08:32:25.950902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:25.950876    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerStarted","Data":"69b38a07751211885993f5b96d98cabf0d53437fa8b7424f79aaf0dc9fdd7fc5"}
Nov 02 08:32:26.021872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:26.021812    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podStartSLOduration=2.021798318 podStartE2EDuration="2.021798318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:26.008798385 +0000 UTC m=+4090.771235292" watchObservedRunningTime="2026-11-02 08:32:26.021798318 +0000 UTC m=+4090.784235223"
Nov 02 08:32:26.281277 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:26.281245    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:26.287293 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:26.287269    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:26.958113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:26.958076    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:26.959099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:26.959074    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:32:30.400042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.399962    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:32:30.400627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.400611    4515 cpu_manager.go:414] "containerMap: removing container" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:30.400681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.400628    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03afe710-f612-4504-9c63-e282b67993f8" containerName="istio-proxy"
Nov 02 08:32:30.404049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.404028    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.432130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.432099    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a44352-dockercfg-8bxnj\""
Nov 02 08:32:30.432130 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.432119    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a44352\""
Nov 02 08:32:30.435462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.435436    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:32:30.467557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.467512    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:32:30.471767 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.471741    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.577385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.577345    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:32:30.611426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.611390    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrtx\" (UniqueName: \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.611712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.611453    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.611712 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.611622    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.611853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.611732    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcxq\" (UniqueName: \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.713640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.713553    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrtx\" (UniqueName: \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.713640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.713601    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.713894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.713683    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.713894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.713793    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcxq\" (UniqueName: \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.714485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.714463    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.714485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.714470    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.752594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.752559    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcxq\" (UniqueName: \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\") pod \"payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:30.767412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.767385    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrtx\" (UniqueName: \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\") pod \"payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:30.879549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:30.879513    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:31.020123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.020087    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:31.199017 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:32:31.198983    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110fce66_0aad_4dee_be59_8e66fd14468d.slice/crio-d4c7119447489ac6987411e226da936c4208fc233425a47e94cbfd18f55566b0 WatchSource:0}: Error finding container d4c7119447489ac6987411e226da936c4208fc233425a47e94cbfd18f55566b0: Status 404 returned error can't find the container with id d4c7119447489ac6987411e226da936c4208fc233425a47e94cbfd18f55566b0
Nov 02 08:32:31.199638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.199603    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:32:31.333111 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:32:31.333074    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52099437_e3e1_4bd8_a9b7_be59c98e14a6.slice/crio-a8e0d5a25eae185a19f889775ab032b82319b20f6f71cf74613d8a67b585342a WatchSource:0}: Error finding container a8e0d5a25eae185a19f889775ab032b82319b20f6f71cf74613d8a67b585342a: Status 404 returned error can't find the container with id a8e0d5a25eae185a19f889775ab032b82319b20f6f71cf74613d8a67b585342a
Nov 02 08:32:31.334411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.334385    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:32:31.993731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.993689    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" event={"ID":"110fce66-0aad-4dee-be59-8e66fd14468d","Type":"ContainerStarted","Data":"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"}
Nov 02 08:32:31.994225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.993794    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" event={"ID":"110fce66-0aad-4dee-be59-8e66fd14468d","Type":"ContainerStarted","Data":"d4c7119447489ac6987411e226da936c4208fc233425a47e94cbfd18f55566b0"}
Nov 02 08:32:31.994225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.993945    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:31.995350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.995291    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" event={"ID":"52099437-e3e1-4bd8-a9b7-be59c98e14a6","Type":"ContainerStarted","Data":"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"}
Nov 02 08:32:31.995494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.995361    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" event={"ID":"52099437-e3e1-4bd8-a9b7-be59c98e14a6","Type":"ContainerStarted","Data":"a8e0d5a25eae185a19f889775ab032b82319b20f6f71cf74613d8a67b585342a"}
Nov 02 08:32:31.995494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:31.995401    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:32.042207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:32.042149    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" podStartSLOduration=2.042136552 podStartE2EDuration="2.042136552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:32.039784664 +0000 UTC m=+4096.802221593" watchObservedRunningTime="2026-11-02 08:32:32.042136552 +0000 UTC m=+4096.804573450"
Nov 02 08:32:32.083969 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:32.083916    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" podStartSLOduration=2.08390303 podStartE2EDuration="2.08390303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:32.083759954 +0000 UTC m=+4096.846196859" watchObservedRunningTime="2026-11-02 08:32:32.08390303 +0000 UTC m=+4096.846339935"
Nov 02 08:32:43.006129 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:43.006082    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:32:43.006576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:43.006398    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:32:44.386405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:32:44.385382    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:32:47.732491 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:32:47.732447    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:33:13.928899 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:13.928863    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:33:17.749986 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:33:17.749944    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:33:24.331894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.331864    4515 generic.go:356] "Generic (PLEG): container finished" podID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerID="77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930" exitCode=0
Nov 02 08:33:24.332423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.331920    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerDied","Data":"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"}
Nov 02 08:33:24.332423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.332369    4515 scope.go:122] "RemoveContainer" containerID="77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"
Nov 02 08:33:24.334036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.334001    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:33:24.334147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.334075    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:33:24.334147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:24.334125    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:33:25.349093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:25.349058    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerStarted","Data":"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"}
Nov 02 08:33:26.281444 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:26.281409    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:33:26.282875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:26.282849    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 02 08:33:26.282953 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:26.282902    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 02 08:33:26.958914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:26.958875    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:33:27.281038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:27.281008    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 02 08:33:27.281230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:27.281058    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 02 08:33:28.282069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:28.282031    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 02 08:33:28.282526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:28.282087    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 02 08:33:29.281117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:29.281083    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 02 08:33:29.281344 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:29.281143    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 02 08:33:30.282194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:30.282160    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Nov 02 08:33:30.282665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:30.282229    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Nov 02 08:33:31.286558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:31.286525    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:33:31.287784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:31.287764    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:33:37.145457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.145420    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:33:37.157992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.157962    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.183325 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.183286    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:33:37.192117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.192091    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-3d8293\"/\"kube-root-ca.crt\""
Nov 02 08:33:37.192117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.192113    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-3d8293\"/\"openshift-service-ca.crt\""
Nov 02 08:33:37.193379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.193362    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-3d8293\"/\"default-dockercfg-ln9lz\""
Nov 02 08:33:37.193703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.193684    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-3d8293\"/\"rate-test-model-3d8293-kserve-self-signed-certs\""
Nov 02 08:33:37.263977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.263933    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.264188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.263989    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.264265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.264233    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.264346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.264279    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5n5v\" (UniqueName: \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.264472 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.264375    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.264472 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.264413    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366119    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366276    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366327    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5n5v\" (UniqueName: \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366407    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366458    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366550    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366566    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366880    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.366903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.366897    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.368675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.368651    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.368982 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.368958    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.398251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.398132    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5n5v\" (UniqueName: \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\") pod \"rate-test-model-3d8293-kserve-c8f547d49-ptj4r\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") " pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.472848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.472809    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:37.779496 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:33:37.779461    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d045dba_a7af_4ff3_988c_332d14f13ce2.slice/crio-25406d3cfb58b7272e459d3c70caaac38b8984ea50c79e357117a80e167458c4 WatchSource:0}: Error finding container 25406d3cfb58b7272e459d3c70caaac38b8984ea50c79e357117a80e167458c4: Status 404 returned error can't find the container with id 25406d3cfb58b7272e459d3c70caaac38b8984ea50c79e357117a80e167458c4
Nov 02 08:33:37.780668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:37.780643    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:33:38.446109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:38.446070    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" event={"ID":"8d045dba-a7af-4ff3-988c-332d14f13ce2","Type":"ContainerStarted","Data":"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"}
Nov 02 08:33:38.446607 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:38.446119    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" event={"ID":"8d045dba-a7af-4ff3-988c-332d14f13ce2","Type":"ContainerStarted","Data":"25406d3cfb58b7272e459d3c70caaac38b8984ea50c79e357117a80e167458c4"}
Nov 02 08:33:38.472899 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:38.472865    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:38.489713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:38.489662    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" podStartSLOduration=1.489649343 podStartE2EDuration="1.489649343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:38.487971378 +0000 UTC m=+4163.250408284" watchObservedRunningTime="2026-11-02 08:33:38.489649343 +0000 UTC m=+4163.252086249"
Nov 02 08:33:46.307662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:46.307622    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:33:47.473696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:47.473657    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:47.486437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:47.486410    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:47.512526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:33:47.512497    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:33:47.723432 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:33:47.723393    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:34:17.723693 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:17.723612    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:34:28.610080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:28.610049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:34:33.325041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.325000    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-6fc1fa"]
Nov 02 08:34:33.329253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.329233    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:33.345851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.345821    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-6fc1fa"]
Nov 02 08:34:33.354456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.354427    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmvw6\""
Nov 02 08:34:33.452046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.452001    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph5l\" (UniqueName: \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\") pod \"test-curl-7361-6fc1fa\" (UID: \"707b4d02-477f-4a19-a477-8e545e5456cf\") " pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:33.553654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.553615    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph5l\" (UniqueName: \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\") pod \"test-curl-7361-6fc1fa\" (UID: \"707b4d02-477f-4a19-a477-8e545e5456cf\") " pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:33.577560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.577474    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph5l\" (UniqueName: \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\") pod \"test-curl-7361-6fc1fa\" (UID: \"707b4d02-477f-4a19-a477-8e545e5456cf\") " pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:33.642216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.642172    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:33.948599 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:34:33.948567    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod707b4d02_477f_4a19_a477_8e545e5456cf.slice/crio-054c3a9275cf914a6308527e06aa1c8c20899feb760973d21b9a72dfe8a89c8e WatchSource:0}: Error finding container 054c3a9275cf914a6308527e06aa1c8c20899feb760973d21b9a72dfe8a89c8e: Status 404 returned error can't find the container with id 054c3a9275cf914a6308527e06aa1c8c20899feb760973d21b9a72dfe8a89c8e
Nov 02 08:34:33.949165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:33.949136    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-6fc1fa"]
Nov 02 08:34:34.816971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:34.816931    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-6fc1fa" event={"ID":"707b4d02-477f-4a19-a477-8e545e5456cf","Type":"ContainerStarted","Data":"9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0"}
Nov 02 08:34:34.817513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:34.816982    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-6fc1fa" event={"ID":"707b4d02-477f-4a19-a477-8e545e5456cf","Type":"ContainerStarted","Data":"054c3a9275cf914a6308527e06aa1c8c20899feb760973d21b9a72dfe8a89c8e"}
Nov 02 08:34:34.858266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:34.858207    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-6fc1fa" podStartSLOduration=1.858191331 podStartE2EDuration="1.858191331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:34.855012509 +0000 UTC m=+4219.617449417" watchObservedRunningTime="2026-11-02 08:34:34.858191331 +0000 UTC m=+4219.620628236"
Nov 02 08:34:35.722198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.722159    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-6fc1fa"]
Nov 02 08:34:35.735587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.735557    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-6fc1fa"]
Nov 02 08:34:35.873730 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.873694    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-8b5248"]
Nov 02 08:34:35.874372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.874349    4515 cpu_manager.go:414] "containerMap: removing container" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" containerName="test-curl-7361-6fc1fa"
Nov 02 08:34:35.874372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.874366    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" containerName="test-curl-7361-6fc1fa"
Nov 02 08:34:35.874486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.874476    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" containerName="test-curl-7361-6fc1fa"
Nov 02 08:34:35.878146 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.878118    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:35.893814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.893791    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-8b5248"]
Nov 02 08:34:35.984454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:35.984347    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjtk\" (UniqueName: \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\") pod \"test-curl-7361-8b5248\" (UID: \"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2\") " pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:36.086042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.086002    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjtk\" (UniqueName: \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\") pod \"test-curl-7361-8b5248\" (UID: \"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2\") " pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:36.109616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.109582    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjtk\" (UniqueName: \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\") pod \"test-curl-7361-8b5248\" (UID: \"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2\") " pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:36.191410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.191369    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:36.489729 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:34:36.489697    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf02058e_4a50_4aef_8dd4_3d6bb6497ff2.slice/crio-ef93752ab06cab650e11315d1a45a4b3c9584ab930264b0b01fba917c4758220 WatchSource:0}: Error finding container ef93752ab06cab650e11315d1a45a4b3c9584ab930264b0b01fba917c4758220: Status 404 returned error can't find the container with id ef93752ab06cab650e11315d1a45a4b3c9584ab930264b0b01fba917c4758220
Nov 02 08:34:36.492524 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.492488    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-8b5248"]
Nov 02 08:34:36.831181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.831091    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-8b5248" event={"ID":"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2","Type":"ContainerStarted","Data":"45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9"}
Nov 02 08:34:36.831181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.831141    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-8b5248" event={"ID":"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2","Type":"ContainerStarted","Data":"ef93752ab06cab650e11315d1a45a4b3c9584ab930264b0b01fba917c4758220"}
Nov 02 08:34:36.831435 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.831227    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-6fc1fa" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" containerName="test-curl-7361-6fc1fa" containerID="cri-o://9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0" gracePeriod=2
Nov 02 08:34:36.845235 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:36.845204    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-6fc1fa\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:36.884121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:36.884068    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-8b5248" podStartSLOduration=1.884055827 podStartE2EDuration="1.884055827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:36.882933351 +0000 UTC m=+4221.645370256" watchObservedRunningTime="2026-11-02 08:34:36.884055827 +0000 UTC m=+4221.646492732"
Nov 02 08:34:36.897079 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:36.897047    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-6fc1fa\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:37.718465 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:37.717417    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-8b5248"]
Nov 02 08:34:37.720610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:37.720573    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-8b5248"]
Nov 02 08:34:38.846421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:38.846380    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-8b5248" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" containerName="test-curl-7361-8b5248" containerID="cri-o://45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9" gracePeriod=2
Nov 02 08:34:38.860150 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:38.860116    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-8b5248\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:39.040580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.040554    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:39.054443 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:39.054403    4515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-8b5248\" is forbidden: User \"system:node:ip-10-0-137-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-156.ec2.internal' and this object" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:39.122062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.121972    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\" (UniqueName: \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\") pod \"707b4d02-477f-4a19-a477-8e545e5456cf\" (UID: \"707b4d02-477f-4a19-a477-8e545e5456cf\") "
Nov 02 08:34:39.124209 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.124179    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l" pod "707b4d02-477f-4a19-a477-8e545e5456cf" (UID: "707b4d02-477f-4a19-a477-8e545e5456cf"). InnerVolumeSpecName "kube-api-access-fph5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:39.224420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.224385    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fph5l\" (UniqueName: \"kubernetes.io/projected/707b4d02-477f-4a19-a477-8e545e5456cf-kube-api-access-fph5l\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:34:39.374891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.374797    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707b4d02-477f-4a19-a477-8e545e5456cf" path="/var/lib/kubelet/pods/707b4d02-477f-4a19-a477-8e545e5456cf/volumes"
Nov 02 08:34:39.855217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.855182    4515 generic.go:356] "Generic (PLEG): container finished" podID="707b4d02-477f-4a19-a477-8e545e5456cf" containerID="9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0" exitCode=137
Nov 02 08:34:39.855700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.855232    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-6fc1fa"
Nov 02 08:34:39.855700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.855285    4515 scope.go:122] "RemoveContainer" containerID="9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0"
Nov 02 08:34:39.867914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.867893    4515 scope.go:122] "RemoveContainer" containerID="9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0"
Nov 02 08:34:39.868257 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:39.868234    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0\": container with ID starting with 9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0 not found: ID does not exist" containerID="9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0"
Nov 02 08:34:39.868365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:39.868264    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0"} err="failed to get container status \"9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0\": rpc error: code = NotFound desc = could not find container \"9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0\": container with ID starting with 9a2faf209890f1af9d1480d766322c9dd97a48478a0bc04be7d911fd2426c3c0 not found: ID does not exist"
Nov 02 08:34:41.064591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.064561    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:41.147191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.147090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\" (UniqueName: \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\") pod \"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2\" (UID: \"cf02058e-4a50-4aef-8dd4-3d6bb6497ff2\") "
Nov 02 08:34:41.149570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.149542    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk" pod "cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" (UID: "cf02058e-4a50-4aef-8dd4-3d6bb6497ff2"). InnerVolumeSpecName "kube-api-access-xqjtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:41.258446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.250063    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjtk\" (UniqueName: \"kubernetes.io/projected/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2-kube-api-access-xqjtk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:34:41.375386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.375349    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" path="/var/lib/kubelet/pods/cf02058e-4a50-4aef-8dd4-3d6bb6497ff2/volumes"
Nov 02 08:34:41.870111 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.870070    4515 generic.go:356] "Generic (PLEG): container finished" podID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" containerID="45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9" exitCode=137
Nov 02 08:34:41.870283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.870126    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-8b5248"
Nov 02 08:34:41.870283 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.870146    4515 scope.go:122] "RemoveContainer" containerID="45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9"
Nov 02 08:34:41.879665 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.879648    4515 scope.go:122] "RemoveContainer" containerID="45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9"
Nov 02 08:34:41.879939 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:41.879922    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9\": container with ID starting with 45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9 not found: ID does not exist" containerID="45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9"
Nov 02 08:34:41.879998 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:41.879952    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9"} err="failed to get container status \"45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9\": rpc error: code = NotFound desc = could not find container \"45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9\": container with ID starting with 45b64fea39713f885cca63cfbbac6988f80166dd92854204c8b7d0338d8bc5c9 not found: ID does not exist"
Nov 02 08:34:47.723886 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:34:47.723848    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:34:54.094930 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:34:54.094889    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:35:03.603154 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:03.603116    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:35:10.680149 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:10.680112    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 08:35:15.681437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:15.681383    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing" containerID="cri-o://b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48" gracePeriod=25
Nov 02 08:35:17.766329 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:17.766274    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:35:18.627841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.627792    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" podUID="0d6b8eda-0315-4668-abed-16075315bd17" containerName="main" containerID="cri-o://f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95" gracePeriod=45
Nov 02 08:35:18.692504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.692429    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" podUID="0d6b8eda-0315-4668-abed-16075315bd17" containerName="main" probeResult="failure" output="Get \"https://10.133.1.49:8000/ready\": dial tcp 10.133.1.49:8000: connect: connection refused"
Nov 02 08:35:18.908942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.908914    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:35:18.963826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.963795    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964019 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.963835    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\" (UniqueName: \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964019 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.963858    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964019 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.963895    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964019 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.963947    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964034    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\") pod \"0d6b8eda-0315-4668-abed-16075315bd17\" (UID: \"0d6b8eda-0315-4668-abed-16075315bd17\") "
Nov 02 08:35:18.964284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964229    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:18.964491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964439    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:18.964491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964456    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:18.964783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964760    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:18.964862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964788    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:18.964862 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.964803    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:18.966177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.966152    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:18.966177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.966166    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:35:18.966431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:18.966408    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9" pod "0d6b8eda-0315-4668-abed-16075315bd17" (UID: "0d6b8eda-0315-4668-abed-16075315bd17"). InnerVolumeSpecName "kube-api-access-klqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:19.066587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.066548    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6b8eda-0315-4668-abed-16075315bd17-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:19.066782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.066592    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klqb9\" (UniqueName: \"kubernetes.io/projected/0d6b8eda-0315-4668-abed-16075315bd17-kube-api-access-klqb9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:19.066782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.066609    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6b8eda-0315-4668-abed-16075315bd17-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:19.116746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.116715    4515 generic.go:356] "Generic (PLEG): container finished" podID="0d6b8eda-0315-4668-abed-16075315bd17" containerID="f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95" exitCode=0
Nov 02 08:35:19.116746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.116756    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"
Nov 02 08:35:19.117076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.116795    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" event={"ID":"0d6b8eda-0315-4668-abed-16075315bd17","Type":"ContainerDied","Data":"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"}
Nov 02 08:35:19.117076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.116839    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb" event={"ID":"0d6b8eda-0315-4668-abed-16075315bd17","Type":"ContainerDied","Data":"6dc6102c60ef9d0d46500ff9bda55e4ba76cfb019ee0acec332440485e954d66"}
Nov 02 08:35:19.117076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.116858    4515 scope.go:122] "RemoveContainer" containerID="f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"
Nov 02 08:35:19.127040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.127015    4515 scope.go:122] "RemoveContainer" containerID="f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"
Nov 02 08:35:19.127379 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:19.127354    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95\": container with ID starting with f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95 not found: ID does not exist" containerID="f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"
Nov 02 08:35:19.127464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.127392    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95"} err="failed to get container status \"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95\": rpc error: code = NotFound desc = could not find container \"f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95\": container with ID starting with f408030ad5c78ba706f1e74331d35ec103e72166e3826428969e8163f5aa9d95 not found: ID does not exist"
Nov 02 08:35:19.180746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.180659    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:35:19.193680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.193572    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56bbb6/rate-test-model-56bbb6-kserve-77f49c7cdf-s2djb"]
Nov 02 08:35:19.375070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:19.375037    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6b8eda-0315-4668-abed-16075315bd17" path="/var/lib/kubelet/pods/0d6b8eda-0315-4668-abed-16075315bd17/volumes"
Nov 02 08:35:21.024594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:21.024559    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:30.041728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:30.041648    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:35:31.025116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:31.025063    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:31.226761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:31.226721    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:35:36.002741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.002714    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 08:35:36.100782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.100743    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\" (UniqueName: \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\") pod \"ed74f42a-49c1-4d00-be11-40e7d9656da4\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") "
Nov 02 08:35:36.100988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.100899    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\") pod \"ed74f42a-49c1-4d00-be11-40e7d9656da4\" (UID: \"ed74f42a-49c1-4d00-be11-40e7d9656da4\") "
Nov 02 08:35:36.101299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.101274    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume" pod "ed74f42a-49c1-4d00-be11-40e7d9656da4" (UID: "ed74f42a-49c1-4d00-be11-40e7d9656da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:36.103043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.103004    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj" pod "ed74f42a-49c1-4d00-be11-40e7d9656da4" (UID: "ed74f42a-49c1-4d00-be11-40e7d9656da4"). InnerVolumeSpecName "kube-api-access-66rvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:36.202144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.202111    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66rvj\" (UniqueName: \"kubernetes.io/projected/ed74f42a-49c1-4d00-be11-40e7d9656da4-kube-api-access-66rvj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:36.202144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.202141    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed74f42a-49c1-4d00-be11-40e7d9656da4-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:36.229196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.229163    4515 generic.go:356] "Generic (PLEG): container finished" podID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerID="b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48" exitCode=0
Nov 02 08:35:36.229440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.229211    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"
Nov 02 08:35:36.229440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.229247    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" event={"ID":"ed74f42a-49c1-4d00-be11-40e7d9656da4","Type":"ContainerDied","Data":"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"}
Nov 02 08:35:36.229440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.229322    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv" event={"ID":"ed74f42a-49c1-4d00-be11-40e7d9656da4","Type":"ContainerDied","Data":"21a67f6fe34dd9f434d9dc6fa2fdb26c1c39d922ca2d574a9aa26112b4bbe9c1"}
Nov 02 08:35:36.229440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.229359    4515 scope.go:122] "RemoveContainer" containerID="b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"
Nov 02 08:35:36.240439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.240420    4515 scope.go:122] "RemoveContainer" containerID="b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"
Nov 02 08:35:36.240730 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:36.240712    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48\": container with ID starting with b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48 not found: ID does not exist" containerID="b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"
Nov 02 08:35:36.240769 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.240739    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48"} err="failed to get container status \"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48\": rpc error: code = NotFound desc = could not find container \"b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48\": container with ID starting with b23e6f23652798414cc0906932d89a9b5d7ca603f15eee5a14f9a5cc2700de48 not found: ID does not exist"
Nov 02 08:35:36.282182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.282149    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 08:35:36.288284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:36.288255    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-696c7a-787b4845b7-dqvxv"]
Nov 02 08:35:37.183365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:37.183302    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 08:35:37.217648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:37.217614    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 08:35:37.374300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:37.374233    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" path="/var/lib/kubelet/pods/ed74f42a-49c1-4d00-be11-40e7d9656da4/volumes"
Nov 02 08:35:42.183781 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.183711    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing" containerID="cri-o://7b6128bf20229a4c0bc21e349f5a13c3027dc23c70963e134d3903a031e1908c" gracePeriod=25
Nov 02 08:35:42.218741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.218689    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing" containerID="cri-o://eede39c256180ce1aeea39dfc6cc5053efb5e125239297894561857c5dbad4b3" gracePeriod=25
Nov 02 08:35:42.801686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.801645    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:35:42.801921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.801894    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy" containerID="cri-o://dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac" gracePeriod=30
Nov 02 08:35:42.814473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.814443    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:35:42.814659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.814637    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy" containerID="cri-o://92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed" gracePeriod=30
Nov 02 08:35:42.817099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.817071    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:35:42.817302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.817271    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy" containerID="cri-o://f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c" gracePeriod=30
Nov 02 08:35:42.827372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.827342    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 08:35:42.827593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:42.827563    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" containerID="cri-o://316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac" gracePeriod=30
Nov 02 08:35:44.259491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:44.259441    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hm547 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:35:44.260012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:44.259505    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:35:46.248215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.248173    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerName="main" containerID="cri-o://f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171" gracePeriod=45
Nov 02 08:35:46.396589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.395845    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerName="main" probeResult="failure" output="Get \"https://10.133.1.38:8000/ready\": dial tcp 10.133.1.38:8000: connect: connection refused"
Nov 02 08:35:46.521902 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.521876    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:35:46.535844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.535818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\" (UniqueName: \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.535987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.535876    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.535987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.535912    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\" (UniqueName: \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.535987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.535935    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.535987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.535956    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.536239 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536016    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\") pod \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\" (UID: \"117b6b36-a1e8-4369-9ae9-e4d2f7299419\") "
Nov 02 08:35:46.536239 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536202    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:46.536239 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536219    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:46.536470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536425    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:46.536807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536786    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:46.536807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536811    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:46.536938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.536828    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:46.538216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.538192    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:35:46.538363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.538217    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "kube-api-access-rw22m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:46.538438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.538391    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm" pod "117b6b36-a1e8-4369-9ae9-e4d2f7299419" (UID: "117b6b36-a1e8-4369-9ae9-e4d2f7299419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:46.638123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.638089    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/117b6b36-a1e8-4369-9ae9-e4d2f7299419-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:46.638123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.638119    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/117b6b36-a1e8-4369-9ae9-e4d2f7299419-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:46.638123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:46.638130    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw22m\" (UniqueName: \"kubernetes.io/projected/117b6b36-a1e8-4369-9ae9-e4d2f7299419-kube-api-access-rw22m\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:47.300929 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.300896    4515 generic.go:356] "Generic (PLEG): container finished" podID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerID="f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171" exitCode=0
Nov 02 08:35:47.301432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.300945    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"
Nov 02 08:35:47.301432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.300985    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" event={"ID":"117b6b36-a1e8-4369-9ae9-e4d2f7299419","Type":"ContainerDied","Data":"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"}
Nov 02 08:35:47.301432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.301022    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf" event={"ID":"117b6b36-a1e8-4369-9ae9-e4d2f7299419","Type":"ContainerDied","Data":"d6510e3d33dd62dda2cabf0fe44acf469d42cb5673c1cfea2a5ceae3c9a05ebb"}
Nov 02 08:35:47.301432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.301038    4515 scope.go:122] "RemoveContainer" containerID="f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"
Nov 02 08:35:47.311060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.311041    4515 scope.go:122] "RemoveContainer" containerID="f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"
Nov 02 08:35:47.311340 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:47.311301    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171\": container with ID starting with f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171 not found: ID does not exist" containerID="f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"
Nov 02 08:35:47.311430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.311351    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171"} err="failed to get container status \"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171\": rpc error: code = NotFound desc = could not find container \"f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171\": container with ID starting with f53990d89c2b9adf9f12bf16dbe4431fde01ab39d70e32a8d21cc4e1744fd171 not found: ID does not exist"
Nov 02 08:35:47.353469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.353433    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:35:47.365863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.365830    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7f19e8/sub-test-model-7f19e8-kserve-6c8f7b8677-7twxf"]
Nov 02 08:35:47.373731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:47.373703    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" path="/var/lib/kubelet/pods/117b6b36-a1e8-4369-9ae9-e4d2f7299419/volumes"
Nov 02 08:35:47.722802 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:47.722718    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:35:48.144804 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.144768    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:35:48.164540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164431    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164491    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164525    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164593    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164639    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164706    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164743    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164830    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.164906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.164902    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\") pod \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\" (UID: \"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1\") "
Nov 02 08:35:48.165889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.165587    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.165889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.165702    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:48.166238 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.166209    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.166962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.166910    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.167068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.166972    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.168715 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.168686    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.171477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.171454    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:35:48.172476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.172437    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "kube-api-access-7hggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.174351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.174271    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token" pod "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" (UID: "ed078af5-0ce7-4c07-9e6f-d4ca95a878d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.242734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.242693    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:35:48.244818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.244796    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 08:35:48.246873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.246851    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:35:48.266071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.265998    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266034    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266067    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266186    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266241    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266274    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266392    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266507    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266403    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266484    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266488    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266549    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266580    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266578    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266658    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266698    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266728    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266756    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266801    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266821    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266870    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266909    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.266987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266959    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.266994    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267027    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267090    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267130    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267161    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\") pod \"daf1506e-87b5-4951-a2fc-64b71d997a0b\" (UID: \"daf1506e-87b5-4951-a2fc-64b71d997a0b\") "
Nov 02 08:35:48.267579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267441    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267587    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267607    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267656    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267697    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\") pod \"ea689304-59b3-46c4-9be7-d12d25a78383\" (UID: \"ea689304-59b3-46c4-9be7-d12d25a78383\") "
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267713    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267814    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267734    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\") pod \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\" (UID: \"0dd38826-5176-4d0c-ba84-d0797d6ed04c\") "
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267884    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.267896    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268090    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268169    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268388    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268641    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268664    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268683    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268698    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268711    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268727    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268741    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268753    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268766    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268782    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268801    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.268813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268817    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268833    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hggf\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-kube-api-access-7hggf\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268847    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268860    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea689304-59b3-46c4-9be7-d12d25a78383-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268876    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268887    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268899    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf1506e-87b5-4951-a2fc-64b71d997a0b-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268912    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268927    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268940    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268954    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268967    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.269609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.268980    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.270279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.269693    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.270472 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.270448    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "kube-api-access-tvwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.270632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.270607    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:35:48.270695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.270638    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "kube-api-access-q6l7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.271260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271221    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token" pod "0dd38826-5176-4d0c-ba84-d0797d6ed04c" (UID: "0dd38826-5176-4d0c-ba84-d0797d6ed04c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.271401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271332    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.271401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271341    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.271401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271359    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:48.271615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271593    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "kube-api-access-mvrnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.271824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.271809    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:35:48.272292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.272277    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo" pod "daf1506e-87b5-4951-a2fc-64b71d997a0b" (UID: "daf1506e-87b5-4951-a2fc-64b71d997a0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:35:48.272368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.272290    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token" pod "ea689304-59b3-46c4-9be7-d12d25a78383" (UID: "ea689304-59b3-46c4-9be7-d12d25a78383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:48.311882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.311842    4515 generic.go:356] "Generic (PLEG): container finished" podID="ea689304-59b3-46c4-9be7-d12d25a78383" containerID="dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac" exitCode=0
Nov 02 08:35:48.312366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.311891    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"
Nov 02 08:35:48.312366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.311920    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" event={"ID":"ea689304-59b3-46c4-9be7-d12d25a78383","Type":"ContainerDied","Data":"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"}
Nov 02 08:35:48.312366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.311977    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5" event={"ID":"ea689304-59b3-46c4-9be7-d12d25a78383","Type":"ContainerDied","Data":"e86cdfdded4a7dc7f6cf3fb88a03d8dd8692dfa78cd5f404003d8bd7c11b8ac5"}
Nov 02 08:35:48.312366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.312007    4515 scope.go:122] "RemoveContainer" containerID="dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"
Nov 02 08:35:48.314810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.314783    4515 generic.go:356] "Generic (PLEG): container finished" podID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerID="92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed" exitCode=0
Nov 02 08:35:48.315226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.314836    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"
Nov 02 08:35:48.315226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.314879    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" event={"ID":"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1","Type":"ContainerDied","Data":"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"}
Nov 02 08:35:48.315226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.314935    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr" event={"ID":"ed078af5-0ce7-4c07-9e6f-d4ca95a878d1","Type":"ContainerDied","Data":"06a839bd7e847784493e79403fc8a7df37d308ed69419021f28aa4f87d1d5a6e"}
Nov 02 08:35:48.316664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.316639    4515 generic.go:356] "Generic (PLEG): container finished" podID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerID="316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac" exitCode=0
Nov 02 08:35:48.316789 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.316700    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"
Nov 02 08:35:48.316789 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.316730    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" event={"ID":"daf1506e-87b5-4951-a2fc-64b71d997a0b","Type":"ContainerDied","Data":"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"}
Nov 02 08:35:48.316945 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.316791    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547" event={"ID":"daf1506e-87b5-4951-a2fc-64b71d997a0b","Type":"ContainerDied","Data":"8021217daf34cbc4bbc59b991a4e0108a9a445a4d54d2ee000375529d9069d30"}
Nov 02 08:35:48.318622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.318394    4515 generic.go:356] "Generic (PLEG): container finished" podID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerID="f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c" exitCode=0
Nov 02 08:35:48.318622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.318436    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"
Nov 02 08:35:48.318622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.318459    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" event={"ID":"0dd38826-5176-4d0c-ba84-d0797d6ed04c","Type":"ContainerDied","Data":"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"}
Nov 02 08:35:48.318622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.318495    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5" event={"ID":"0dd38826-5176-4d0c-ba84-d0797d6ed04c","Type":"ContainerDied","Data":"d1c626fee18e69c9f4f3ac90e892895b86f587c31d163fd67561e9e439966680"}
Nov 02 08:35:48.326546 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.326447    4515 scope.go:122] "RemoveContainer" containerID="dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"
Nov 02 08:35:48.329669 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:48.329611    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac\": container with ID starting with dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac not found: ID does not exist" containerID="dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"
Nov 02 08:35:48.329669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.329651    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac"} err="failed to get container status \"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac\": rpc error: code = NotFound desc = could not find container \"dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac\": container with ID starting with dcc7c3d0430cfe49cc35450e00044e769fed90e3f8a585b74edcf6a6217886ac not found: ID does not exist"
Nov 02 08:35:48.329669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.329672    4515 scope.go:122] "RemoveContainer" containerID="92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"
Nov 02 08:35:48.344445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.344423    4515 scope.go:122] "RemoveContainer" containerID="92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"
Nov 02 08:35:48.344781 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:48.344761    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed\": container with ID starting with 92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed not found: ID does not exist" containerID="92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"
Nov 02 08:35:48.344878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.344797    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed"} err="failed to get container status \"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed\": rpc error: code = NotFound desc = could not find container \"92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed\": container with ID starting with 92f5febba2a51dc93227b3d5a5d2169bc4d86fe16c091997ca06d5fc3d3844ed not found: ID does not exist"
Nov 02 08:35:48.344878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.344821    4515 scope.go:122] "RemoveContainer" containerID="316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"
Nov 02 08:35:48.356694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.356671    4515 scope.go:122] "RemoveContainer" containerID="316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"
Nov 02 08:35:48.357036 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:48.357011    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac\": container with ID starting with 316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac not found: ID does not exist" containerID="316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"
Nov 02 08:35:48.357090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.357046    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac"} err="failed to get container status \"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac\": rpc error: code = NotFound desc = could not find container \"316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac\": container with ID starting with 316f53f11c01b9c646941cf904a6c77c792aa0d72df1355fae2d4d46d1a14fac not found: ID does not exist"
Nov 02 08:35:48.357090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.357074    4515 scope.go:122] "RemoveContainer" containerID="f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"
Nov 02 08:35:48.366723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.366701    4515 scope.go:122] "RemoveContainer" containerID="f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"
Nov 02 08:35:48.367074 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:35:48.367047    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c\": container with ID starting with f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c not found: ID does not exist" containerID="f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"
Nov 02 08:35:48.367138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.367086    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c"} err="failed to get container status \"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c\": rpc error: code = NotFound desc = could not find container \"f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c\": container with ID starting with f678c79820705a22106278ddb67391cbf619c2d9cff6717d2f5f875c5485875c not found: ID does not exist"
Nov 02 08:35:48.370014 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.369989    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea689304-59b3-46c4-9be7-d12d25a78383-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370023    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370039    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370054    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370068    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6l7t\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-kube-api-access-q6l7t\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370094    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwxz\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-kube-api-access-tvwxz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370107    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea689304-59b3-46c4-9be7-d12d25a78383-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370120    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvrnj\" (UniqueName: \"kubernetes.io/projected/daf1506e-87b5-4951-a2fc-64b71d997a0b-kube-api-access-mvrnj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370133    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea689304-59b3-46c4-9be7-d12d25a78383-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370148    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd38826-5176-4d0c-ba84-d0797d6ed04c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370160    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.370405 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.370173    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf1506e-87b5-4951-a2fc-64b71d997a0b-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:35:48.425996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.425960    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:35:48.428679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.428649    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hf7z5"]
Nov 02 08:35:48.458684 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.458648    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:35:48.463439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.463413    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gppb5"]
Nov 02 08:35:48.499909 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.499873    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:35:48.504829 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.504781    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-n97lr"]
Nov 02 08:35:48.541374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.541272    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 08:35:48.547015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.546982    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hm547"]
Nov 02 08:35:48.900810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.900755    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:48.901213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:48.901182    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:49.373003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:49.372970    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" path="/var/lib/kubelet/pods/0dd38826-5176-4d0c-ba84-d0797d6ed04c/volumes"
Nov 02 08:35:49.373567 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:49.373551    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" path="/var/lib/kubelet/pods/daf1506e-87b5-4951-a2fc-64b71d997a0b/volumes"
Nov 02 08:35:49.373978 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:49.373965    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" path="/var/lib/kubelet/pods/ea689304-59b3-46c4-9be7-d12d25a78383/volumes"
Nov 02 08:35:49.374692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:49.374676    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" path="/var/lib/kubelet/pods/ed078af5-0ce7-4c07-9e6f-d4ca95a878d1/volumes"
Nov 02 08:35:58.894643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:58.894604    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:58.895040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:58.894614    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:59.937038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:35:59.936990    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:36:02.411754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.411721    4515 generic.go:356] "Generic (PLEG): container finished" podID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerID="7b6128bf20229a4c0bc21e349f5a13c3027dc23c70963e134d3903a031e1908c" exitCode=0
Nov 02 08:36:02.412363 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.411764    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" event={"ID":"ec4d2d48-36c3-4144-9b19-a60aa603d91d","Type":"ContainerDied","Data":"7b6128bf20229a4c0bc21e349f5a13c3027dc23c70963e134d3903a031e1908c"}
Nov 02 08:36:02.413662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.413635    4515 generic.go:356] "Generic (PLEG): container finished" podID="73f35d01-275a-4261-a631-8d0e008f8760" containerID="eede39c256180ce1aeea39dfc6cc5053efb5e125239297894561857c5dbad4b3" exitCode=0
Nov 02 08:36:02.413782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.413706    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" event={"ID":"73f35d01-275a-4261-a631-8d0e008f8760","Type":"ContainerDied","Data":"eede39c256180ce1aeea39dfc6cc5053efb5e125239297894561857c5dbad4b3"}
Nov 02 08:36:02.490600 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.490573    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 08:36:02.512943 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.512918    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 08:36:02.637566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.637532    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\") pod \"73f35d01-275a-4261-a631-8d0e008f8760\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") "
Nov 02 08:36:02.637785 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.637591    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\" (UniqueName: \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\") pod \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") "
Nov 02 08:36:02.637905 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.637870    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\") pod \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\" (UID: \"ec4d2d48-36c3-4144-9b19-a60aa603d91d\") "
Nov 02 08:36:02.638055 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.637935    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\" (UniqueName: \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\") pod \"73f35d01-275a-4261-a631-8d0e008f8760\" (UID: \"73f35d01-275a-4261-a631-8d0e008f8760\") "
Nov 02 08:36:02.638167 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.638143    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume" pod "73f35d01-275a-4261-a631-8d0e008f8760" (UID: "73f35d01-275a-4261-a631-8d0e008f8760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:02.638258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.638235    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume" pod "ec4d2d48-36c3-4144-9b19-a60aa603d91d" (UID: "ec4d2d48-36c3-4144-9b19-a60aa603d91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:02.638498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.638477    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f35d01-275a-4261-a631-8d0e008f8760-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:02.638568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.638503    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d2d48-36c3-4144-9b19-a60aa603d91d-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:02.639870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.639850    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f" pod "ec4d2d48-36c3-4144-9b19-a60aa603d91d" (UID: "ec4d2d48-36c3-4144-9b19-a60aa603d91d"). InnerVolumeSpecName "kube-api-access-4rl5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:02.639951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.639937    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl" pod "73f35d01-275a-4261-a631-8d0e008f8760" (UID: "73f35d01-275a-4261-a631-8d0e008f8760"). InnerVolumeSpecName "kube-api-access-fhvpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:02.740041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.739951    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvpl\" (UniqueName: \"kubernetes.io/projected/73f35d01-275a-4261-a631-8d0e008f8760-kube-api-access-fhvpl\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:02.740041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:02.739986    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rl5f\" (UniqueName: \"kubernetes.io/projected/ec4d2d48-36c3-4144-9b19-a60aa603d91d-kube-api-access-4rl5f\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.422560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.422531    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"
Nov 02 08:36:03.423032 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.422563    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc" event={"ID":"73f35d01-275a-4261-a631-8d0e008f8760","Type":"ContainerDied","Data":"384b8f6d8dda740b777e90c436f00ade5b4a230aa6eef8736561b51026f672b3"}
Nov 02 08:36:03.423032 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.422624    4515 scope.go:122] "RemoveContainer" containerID="eede39c256180ce1aeea39dfc6cc5053efb5e125239297894561857c5dbad4b3"
Nov 02 08:36:03.424084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.424062    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb" event={"ID":"ec4d2d48-36c3-4144-9b19-a60aa603d91d","Type":"ContainerDied","Data":"471672e0da690abc8cc2f5482035fe06aa3177d39d907e3731ecdd068c5740cd"}
Nov 02 08:36:03.424162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.424084    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"
Nov 02 08:36:03.432519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.432456    4515 scope.go:122] "RemoveContainer" containerID="7b6128bf20229a4c0bc21e349f5a13c3027dc23c70963e134d3903a031e1908c"
Nov 02 08:36:03.478194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.478106    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 08:36:03.490512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.490478    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c712b1-6c465d59f9-ctsxc"]
Nov 02 08:36:03.517434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.517395    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 08:36:03.521096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:03.521068    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c712b1-f8c68f96c-jwpdb"]
Nov 02 08:36:05.373132 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:05.373089    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f35d01-275a-4261-a631-8d0e008f8760" path="/var/lib/kubelet/pods/73f35d01-275a-4261-a631-8d0e008f8760/volumes"
Nov 02 08:36:05.373557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:05.373503    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" path="/var/lib/kubelet/pods/ec4d2d48-36c3-4144-9b19-a60aa603d91d/volumes"
Nov 02 08:36:05.565499 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:05.565466    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 08:36:05.600386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:05.600349    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 08:36:10.566384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:10.566341    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing" containerID="cri-o://a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444" gracePeriod=25
Nov 02 08:36:10.600845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:10.600796    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing" containerID="cri-o://ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5" gracePeriod=25
Nov 02 08:36:10.935350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:10.935226    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:10.935350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:10.935275    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:17.723851 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:17.723810    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:36:20.935520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:20.935470    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:20.935948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:20.935552    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:22.887351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:22.887298    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 08:36:22.888275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:22.888242    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" podUID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" containerName="maas-api" containerID="cri-o://91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da" gracePeriod=30
Nov 02 08:36:23.192827 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.192793    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 08:36:23.268627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.268593    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\") pod \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") "
Nov 02 08:36:23.268791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.268648    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\" (UniqueName: \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\") pod \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") "
Nov 02 08:36:23.268791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.268720    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\") pod \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") "
Nov 02 08:36:23.268963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.268802    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\") pod \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\" (UID: \"ec7be5b5-9c26-43fd-9026-06dcc11ce70f\") "
Nov 02 08:36:23.269271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.269236    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca" pod "ec7be5b5-9c26-43fd-9026-06dcc11ce70f" (UID: "ec7be5b5-9c26-43fd-9026-06dcc11ce70f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:23.269421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.269403    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:23.271059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.271026    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs" pod "ec7be5b5-9c26-43fd-9026-06dcc11ce70f" (UID: "ec7be5b5-9c26-43fd-9026-06dcc11ce70f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:23.271166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.271079    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp" pod "ec7be5b5-9c26-43fd-9026-06dcc11ce70f" (UID: "ec7be5b5-9c26-43fd-9026-06dcc11ce70f"). InnerVolumeSpecName "kube-api-access-n8zlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:23.271424 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.271374    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls" pod "ec7be5b5-9c26-43fd-9026-06dcc11ce70f" (UID: "ec7be5b5-9c26-43fd-9026-06dcc11ce70f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:23.370643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.370519    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:23.370643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.370556    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:23.370643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.370571    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8zlp\" (UniqueName: \"kubernetes.io/projected/ec7be5b5-9c26-43fd-9026-06dcc11ce70f-kube-api-access-n8zlp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:23.545614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.545574    4515 generic.go:356] "Generic (PLEG): container finished" podID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" containerID="91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da" exitCode=0
Nov 02 08:36:23.545808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.545626    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"
Nov 02 08:36:23.545808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.545664    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" event={"ID":"ec7be5b5-9c26-43fd-9026-06dcc11ce70f","Type":"ContainerDied","Data":"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"}
Nov 02 08:36:23.545808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.545731    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp" event={"ID":"ec7be5b5-9c26-43fd-9026-06dcc11ce70f","Type":"ContainerDied","Data":"45452f64adf631b3b5fb41a3e18a4bcc81f647f0a9b476940ba16bba02dac0b4"}
Nov 02 08:36:23.545808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.545759    4515 scope.go:122] "RemoveContainer" containerID="91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"
Nov 02 08:36:23.555407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.555385    4515 scope.go:122] "RemoveContainer" containerID="91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"
Nov 02 08:36:23.555696 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:23.555674    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da\": container with ID starting with 91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da not found: ID does not exist" containerID="91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"
Nov 02 08:36:23.555749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.555704    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da"} err="failed to get container status \"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da\": rpc error: code = NotFound desc = could not find container \"91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da\": container with ID starting with 91653f51f2a2ebce948dfe491d6123903fe1e8d36779fad40407646a5a01f6da not found: ID does not exist"
Nov 02 08:36:23.613894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.613858    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 08:36:23.619194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:23.619161    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-12df3c-75f4b8cf5f-cdvtp"]
Nov 02 08:36:25.374662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:25.374630    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" path="/var/lib/kubelet/pods/ec7be5b5-9c26-43fd-9026-06dcc11ce70f/volumes"
Nov 02 08:36:27.841266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:27.841226    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:36:28.933632 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:28.933578    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:36:29.639128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:29.639093    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 08:36:29.639420 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:29.639388    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy" containerID="cri-o://293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48" gracePeriod=30
Nov 02 08:36:30.879493 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.879467    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 08:36:30.911121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.911097    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 08:36:30.962784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.962747    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\" (UniqueName: \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\") pod \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") "
Nov 02 08:36:30.962997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.962834    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\") pod \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\" (UID: \"c35ac2e0-4ee8-4752-9a1f-51229b40e979\") "
Nov 02 08:36:30.963705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.963659    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume" pod "c35ac2e0-4ee8-4752-9a1f-51229b40e979" (UID: "c35ac2e0-4ee8-4752-9a1f-51229b40e979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:30.966056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:30.966026    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f" pod "c35ac2e0-4ee8-4752-9a1f-51229b40e979" (UID: "c35ac2e0-4ee8-4752-9a1f-51229b40e979"). InnerVolumeSpecName "kube-api-access-9q86f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:31.064557 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.064525    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\") pod \"7a79ad55-c933-453f-8924-254ec0d0011e\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") "
Nov 02 08:36:31.064770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.064620    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\" (UniqueName: \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\") pod \"7a79ad55-c933-453f-8924-254ec0d0011e\" (UID: \"7a79ad55-c933-453f-8924-254ec0d0011e\") "
Nov 02 08:36:31.064994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.064894    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume" pod "7a79ad55-c933-453f-8924-254ec0d0011e" (UID: "7a79ad55-c933-453f-8924-254ec0d0011e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:31.065061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.065036    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q86f\" (UniqueName: \"kubernetes.io/projected/c35ac2e0-4ee8-4752-9a1f-51229b40e979-kube-api-access-9q86f\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:31.065116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.065057    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35ac2e0-4ee8-4752-9a1f-51229b40e979-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:31.066797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.066774    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k" pod "7a79ad55-c933-453f-8924-254ec0d0011e" (UID: "7a79ad55-c933-453f-8924-254ec0d0011e"). InnerVolumeSpecName "kube-api-access-fw76k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:31.166817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.166736    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a79ad55-c933-453f-8924-254ec0d0011e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:31.166817 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.166764    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw76k\" (UniqueName: \"kubernetes.io/projected/7a79ad55-c933-453f-8924-254ec0d0011e-kube-api-access-fw76k\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:31.595656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.595627    4515 generic.go:356] "Generic (PLEG): container finished" podID="7a79ad55-c933-453f-8924-254ec0d0011e" containerID="ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5" exitCode=0
Nov 02 08:36:31.595848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.595671    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"
Nov 02 08:36:31.595848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.595686    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" event={"ID":"7a79ad55-c933-453f-8924-254ec0d0011e","Type":"ContainerDied","Data":"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"}
Nov 02 08:36:31.595848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.595754    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8" event={"ID":"7a79ad55-c933-453f-8924-254ec0d0011e","Type":"ContainerDied","Data":"42eb9b3e0d687152cb13eecafe97ff76e7df9e658b4e85a39610a5463cb1cbbc"}
Nov 02 08:36:31.595848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.595784    4515 scope.go:122] "RemoveContainer" containerID="ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"
Nov 02 08:36:31.597329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.597285    4515 generic.go:356] "Generic (PLEG): container finished" podID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerID="a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444" exitCode=0
Nov 02 08:36:31.597445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.597353    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"
Nov 02 08:36:31.597516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.597444    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" event={"ID":"c35ac2e0-4ee8-4752-9a1f-51229b40e979","Type":"ContainerDied","Data":"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"}
Nov 02 08:36:31.597516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.597497    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p" event={"ID":"c35ac2e0-4ee8-4752-9a1f-51229b40e979","Type":"ContainerDied","Data":"d18b9f36d7c8c63a986107c23d5fc943304ff69297e1f01cbd0b9e6c53f7c4ac"}
Nov 02 08:36:31.608272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.608251    4515 scope.go:122] "RemoveContainer" containerID="ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"
Nov 02 08:36:31.608885 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:31.608855    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5\": container with ID starting with ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5 not found: ID does not exist" containerID="ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"
Nov 02 08:36:31.609014 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.608893    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5"} err="failed to get container status \"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5\": rpc error: code = NotFound desc = could not find container \"ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5\": container with ID starting with ecb935868058475d408c1598deed9d020878f0e5538989511a1a22c647bcfeb5 not found: ID does not exist"
Nov 02 08:36:31.609014 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.608915    4515 scope.go:122] "RemoveContainer" containerID="a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"
Nov 02 08:36:31.617694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.617676    4515 scope.go:122] "RemoveContainer" containerID="a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"
Nov 02 08:36:31.617961 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:31.617940    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444\": container with ID starting with a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444 not found: ID does not exist" containerID="a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"
Nov 02 08:36:31.618010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.617967    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444"} err="failed to get container status \"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444\": rpc error: code = NotFound desc = could not find container \"a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444\": container with ID starting with a6c5deeffb2309b1ce3c58aa4a451366e85af43edd30811bb974aa8f20c91444 not found: ID does not exist"
Nov 02 08:36:31.651928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.651900    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 08:36:31.663324 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.663287    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-12df3c-8f7c97f8f-wgk2p"]
Nov 02 08:36:31.693350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.693173    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 08:36:31.696644 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:31.696618    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12df3c-69559b4d55-tx9t8"]
Nov 02 08:36:33.377298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:33.377257    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" path="/var/lib/kubelet/pods/7a79ad55-c933-453f-8924-254ec0d0011e/volumes"
Nov 02 08:36:33.377987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:33.377955    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" path="/var/lib/kubelet/pods/c35ac2e0-4ee8-4752-9a1f-51229b40e979/volumes"
Nov 02 08:36:34.899623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:34.899591    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 08:36:35.009510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009476    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009523    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009546    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009638    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009694    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009729    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009772    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009807    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.009942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009837    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:35.009942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.009855    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\") pod \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\" (UID: \"6959f0f6-a227-47c5-a6e0-c445d7fc35e2\") "
Nov 02 08:36:35.010241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010008    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:35.010241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010057    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:35.010241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010232    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:35.010471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010349    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:35.010471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010398    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.010471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010415    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.010471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010425    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.010471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.010434    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.012034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.012009    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:35.012034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.012015    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:35.012380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.012363    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:35.012454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.012386    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr" pod "6959f0f6-a227-47c5-a6e0-c445d7fc35e2" (UID: "6959f0f6-a227-47c5-a6e0-c445d7fc35e2"). InnerVolumeSpecName "kube-api-access-4q2cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:35.111892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.111852    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q2cr\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-kube-api-access-4q2cr\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.111892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.111887    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.111892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.111897    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.112169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.111908    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.112169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.111917    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6959f0f6-a227-47c5-a6e0-c445d7fc35e2-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:35.625788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.625751    4515 generic.go:356] "Generic (PLEG): container finished" podID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerID="293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48" exitCode=0
Nov 02 08:36:35.625967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.625805    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"
Nov 02 08:36:35.625967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.625837    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerDied","Data":"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"}
Nov 02 08:36:35.625967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.625899    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd" event={"ID":"6959f0f6-a227-47c5-a6e0-c445d7fc35e2","Type":"ContainerDied","Data":"887cba62c46157b46a09849f84c9352438513bd58d32948bffb4a4b645448621"}
Nov 02 08:36:35.625967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.625929    4515 scope.go:122] "RemoveContainer" containerID="293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"
Nov 02 08:36:35.635414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.635377    4515 scope.go:122] "RemoveContainer" containerID="7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"
Nov 02 08:36:35.644418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.644394    4515 scope.go:122] "RemoveContainer" containerID="293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"
Nov 02 08:36:35.644686 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:35.644651    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48\": container with ID starting with 293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48 not found: ID does not exist" containerID="293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"
Nov 02 08:36:35.644686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.644681    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48"} err="failed to get container status \"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48\": rpc error: code = NotFound desc = could not find container \"293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48\": container with ID starting with 293b395d266c04ed8e18d5f5c85061e693506a1c549f7e0a1e4ba969d3f39f48 not found: ID does not exist"
Nov 02 08:36:35.644852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.644699    4515 scope.go:122] "RemoveContainer" containerID="7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"
Nov 02 08:36:35.645023 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:35.644997    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c\": container with ID starting with 7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c not found: ID does not exist" containerID="7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"
Nov 02 08:36:35.645081 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.645036    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c"} err="failed to get container status \"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c\": rpc error: code = NotFound desc = could not find container \"7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c\": container with ID starting with 7861ae6206446c0a78af82dfdfda35409edbc017a8c6cc1ac78230a54f780d9c not found: ID does not exist"
Nov 02 08:36:35.692227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.692192    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 08:36:35.705203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:35.705173    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-12df3c-openshift-default-6dd7fc4b59-h8whd"]
Nov 02 08:36:37.373007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:37.372972    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" path="/var/lib/kubelet/pods/6959f0f6-a227-47c5-a6e0-c445d7fc35e2/volumes"
Nov 02 08:36:40.531852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:40.531749    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 08:36:42.862502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:42.862460    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" podUID="8d045dba-a7af-4ff3-988c-332d14f13ce2" containerName="main" containerID="cri-o://4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f" gracePeriod=45
Nov 02 08:36:43.122200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.122140    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:36:43.199761 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199729    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.199931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199775    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\" (UniqueName: \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.199931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.199931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199843    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.199931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199872    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.200192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.199947    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\") pod \"8d045dba-a7af-4ff3-988c-332d14f13ce2\" (UID: \"8d045dba-a7af-4ff3-988c-332d14f13ce2\") "
Nov 02 08:36:43.200192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200067    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:43.200192 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200084    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:43.200395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200232    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:43.200472 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200449    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.200539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200475    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.200539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.200489    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.202386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.202363    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:43.202527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.202391    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "kube-api-access-m5n5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:43.202527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.202403    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs" pod "8d045dba-a7af-4ff3-988c-332d14f13ce2" (UID: "8d045dba-a7af-4ff3-988c-332d14f13ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:43.301962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.301921    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5n5v\" (UniqueName: \"kubernetes.io/projected/8d045dba-a7af-4ff3-988c-332d14f13ce2-kube-api-access-m5n5v\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.301962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.301956    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d045dba-a7af-4ff3-988c-332d14f13ce2-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.301962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.301965    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d045dba-a7af-4ff3-988c-332d14f13ce2-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:43.674881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.674791    4515 generic.go:356] "Generic (PLEG): container finished" podID="8d045dba-a7af-4ff3-988c-332d14f13ce2" containerID="4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f" exitCode=0
Nov 02 08:36:43.674881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.674838    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"
Nov 02 08:36:43.675068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.674879    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" event={"ID":"8d045dba-a7af-4ff3-988c-332d14f13ce2","Type":"ContainerDied","Data":"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"}
Nov 02 08:36:43.675068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.674924    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r" event={"ID":"8d045dba-a7af-4ff3-988c-332d14f13ce2","Type":"ContainerDied","Data":"25406d3cfb58b7272e459d3c70caaac38b8984ea50c79e357117a80e167458c4"}
Nov 02 08:36:43.675068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.674943    4515 scope.go:122] "RemoveContainer" containerID="4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"
Nov 02 08:36:43.684269 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.684250    4515 scope.go:122] "RemoveContainer" containerID="4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"
Nov 02 08:36:43.684571 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:43.684553    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f\": container with ID starting with 4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f not found: ID does not exist" containerID="4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"
Nov 02 08:36:43.684634 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.684580    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f"} err="failed to get container status \"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f\": rpc error: code = NotFound desc = could not find container \"4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f\": container with ID starting with 4395d2792ccea2126cb4386ec4c54c7279cf0c415497f72fb966b23641401f0f not found: ID does not exist"
Nov 02 08:36:43.726701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.726667    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:36:43.739416 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:43.739383    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-3d8293/rate-test-model-3d8293-kserve-c8f547d49-ptj4r"]
Nov 02 08:36:44.551773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:44.551740    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:36:45.056503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:45.056472    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:36:45.090936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:45.090895    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:36:45.374836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:45.374743    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d045dba-a7af-4ff3-988c-332d14f13ce2" path="/var/lib/kubelet/pods/8d045dba-a7af-4ff3-988c-332d14f13ce2/volumes"
Nov 02 08:36:45.532689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:45.532644    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing" containerID="cri-o://27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81" gracePeriod=25
Nov 02 08:36:47.723437 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:47.723388    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:36:47.872894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:47.872854    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:49.208526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.208488    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 08:36:49.209266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.208759    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" podUID="e54c8c31-e110-431c-8233-7e7222e45ddf" containerName="maas-api" containerID="cri-o://6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33" gracePeriod=30
Nov 02 08:36:49.489407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.489376    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 08:36:49.568298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.568260    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\") pod \"e54c8c31-e110-431c-8233-7e7222e45ddf\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") "
Nov 02 08:36:49.568522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.568327    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\") pod \"e54c8c31-e110-431c-8233-7e7222e45ddf\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") "
Nov 02 08:36:49.568522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.568414    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\" (UniqueName: \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\") pod \"e54c8c31-e110-431c-8233-7e7222e45ddf\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") "
Nov 02 08:36:49.568522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.568446    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\") pod \"e54c8c31-e110-431c-8233-7e7222e45ddf\" (UID: \"e54c8c31-e110-431c-8233-7e7222e45ddf\") "
Nov 02 08:36:49.569083 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.569049    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca" pod "e54c8c31-e110-431c-8233-7e7222e45ddf" (UID: "e54c8c31-e110-431c-8233-7e7222e45ddf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:49.574341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.571285    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls" pod "e54c8c31-e110-431c-8233-7e7222e45ddf" (UID: "e54c8c31-e110-431c-8233-7e7222e45ddf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:49.574341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.571455    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs" pod "e54c8c31-e110-431c-8233-7e7222e45ddf" (UID: "e54c8c31-e110-431c-8233-7e7222e45ddf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:49.574341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.572437    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd" pod "e54c8c31-e110-431c-8233-7e7222e45ddf" (UID: "e54c8c31-e110-431c-8233-7e7222e45ddf"). InnerVolumeSpecName "kube-api-access-2fvhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:49.670388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.670342    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvhd\" (UniqueName: \"kubernetes.io/projected/e54c8c31-e110-431c-8233-7e7222e45ddf-kube-api-access-2fvhd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:49.670388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.670379    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:49.670388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.670389    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e54c8c31-e110-431c-8233-7e7222e45ddf-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:49.670388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.670399    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e54c8c31-e110-431c-8233-7e7222e45ddf-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:49.716662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.716563    4515 generic.go:356] "Generic (PLEG): container finished" podID="e54c8c31-e110-431c-8233-7e7222e45ddf" containerID="6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33" exitCode=0
Nov 02 08:36:49.716662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.716615    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"
Nov 02 08:36:49.716662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.716648    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" event={"ID":"e54c8c31-e110-431c-8233-7e7222e45ddf","Type":"ContainerDied","Data":"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"}
Nov 02 08:36:49.716984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.716716    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj" event={"ID":"e54c8c31-e110-431c-8233-7e7222e45ddf","Type":"ContainerDied","Data":"2cefc161c593b358d9363b219beb5ae8bf80cf2dc42af062008cc4bf4d9de9bc"}
Nov 02 08:36:49.716984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.716741    4515 scope.go:122] "RemoveContainer" containerID="6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"
Nov 02 08:36:49.728097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.728076    4515 scope.go:122] "RemoveContainer" containerID="6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"
Nov 02 08:36:49.728359 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:49.728332    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33\": container with ID starting with 6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33 not found: ID does not exist" containerID="6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"
Nov 02 08:36:49.728423 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.728371    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33"} err="failed to get container status \"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33\": rpc error: code = NotFound desc = could not find container \"6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33\": container with ID starting with 6451f38711fe461aec32090133eb2bcd8ae7b197602695f5a54ed565ad17be33 not found: ID does not exist"
Nov 02 08:36:49.768598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.768566    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 08:36:49.772968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:49.772943    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56bbb6-757b86958b-b56hj"]
Nov 02 08:36:50.057805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:50.057769    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing" containerID="cri-o://0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817" gracePeriod=25
Nov 02 08:36:50.091825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:50.091779    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing" containerID="cri-o://bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862" gracePeriod=25
Nov 02 08:36:51.384292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:51.384255    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54c8c31-e110-431c-8233-7e7222e45ddf" path="/var/lib/kubelet/pods/e54c8c31-e110-431c-8233-7e7222e45ddf/volumes"
Nov 02 08:36:51.423749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:51.423716    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 08:36:51.423961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:51.423935    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" containerID="cri-o://f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954" gracePeriod=30
Nov 02 08:36:56.087565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.087530    4515 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:36:56.087962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.087586    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:36:56.679560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.679534    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 08:36:56.747911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.747831    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.747911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.747876    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.747911 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.747903    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.747979    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748058    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748099    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748173    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748185    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:56.748216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748187    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748294    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748392    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:56.748568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748411    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:56.748568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748417    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\") pod \"d5237e1b-4892-48aa-8b09-d06e90fbd505\" (UID: \"d5237e1b-4892-48aa-8b09-d06e90fbd505\") "
Nov 02 08:36:56.748797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748718    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:56.749007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.748985    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.749104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.749012    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5237e1b-4892-48aa-8b09-d06e90fbd505-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.749104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.749025    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.749104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.749037    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.749104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.749049    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.750873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.750844    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:56.751013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.750889    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:56.751177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.751147    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:56.751479 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.751459    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb" pod "d5237e1b-4892-48aa-8b09-d06e90fbd505" (UID: "d5237e1b-4892-48aa-8b09-d06e90fbd505"). InnerVolumeSpecName "kube-api-access-pdsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:56.760384 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.760361    4515 generic.go:356] "Generic (PLEG): container finished" podID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerID="f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954" exitCode=0
Nov 02 08:36:56.760514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.760421    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerDied","Data":"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"}
Nov 02 08:36:56.760514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.760425    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"
Nov 02 08:36:56.760514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.760465    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd" event={"ID":"d5237e1b-4892-48aa-8b09-d06e90fbd505","Type":"ContainerDied","Data":"e7195b6d607f28493747d5d3dacfed4bc5e85fba9209214d6e41f9de93d8ee39"}
Nov 02 08:36:56.760514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.760488    4515 scope.go:122] "RemoveContainer" containerID="f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"
Nov 02 08:36:56.776175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.776157    4515 scope.go:122] "RemoveContainer" containerID="548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"
Nov 02 08:36:56.785333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.785294    4515 scope.go:122] "RemoveContainer" containerID="f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"
Nov 02 08:36:56.785644 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:56.785621    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954\": container with ID starting with f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954 not found: ID does not exist" containerID="f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"
Nov 02 08:36:56.785708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.785654    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954"} err="failed to get container status \"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954\": rpc error: code = NotFound desc = could not find container \"f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954\": container with ID starting with f71219f2fa8440a36ddef157b6ebece55685aecd365f154450770ed22e13a954 not found: ID does not exist"
Nov 02 08:36:56.785708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.785674    4515 scope.go:122] "RemoveContainer" containerID="548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"
Nov 02 08:36:56.785957 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:56.785932    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b\": container with ID starting with 548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b not found: ID does not exist" containerID="548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"
Nov 02 08:36:56.786046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.785960    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b"} err="failed to get container status \"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b\": rpc error: code = NotFound desc = could not find container \"548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b\": container with ID starting with 548ac6ac5a42546f0fdff91f94e691f159d0881f58618955ae46a42d0aaf136b not found: ID does not exist"
Nov 02 08:36:56.815583 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.815548    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 08:36:56.819480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.819449    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56bbb6-openshift-default-775f4bf5c-2kwtd"]
Nov 02 08:36:56.850671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.850639    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.850671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.850667    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.850853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.850678    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdsqb\" (UniqueName: \"kubernetes.io/projected/d5237e1b-4892-48aa-8b09-d06e90fbd505-kube-api-access-pdsqb\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.850853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.850689    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5237e1b-4892-48aa-8b09-d06e90fbd505-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.991740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:56.991705    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 08:36:57.036651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.036533    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 08:36:57.038056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.038019    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" podUID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" containerName="maas-api" containerID="cri-o://4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4" gracePeriod=30
Nov 02 08:36:57.278260 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.278229    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 08:36:57.356156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356074    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\") pod \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") "
Nov 02 08:36:57.356156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356149    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\" (UniqueName: \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\") pod \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") "
Nov 02 08:36:57.356392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356281    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\") pod \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") "
Nov 02 08:36:57.356392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356376    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\") pod \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\" (UID: \"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d\") "
Nov 02 08:36:57.356753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356717    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca" pod "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" (UID: "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:57.356904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.356883    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:57.358526 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.358497    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs" pod "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" (UID: "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:57.358636 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.358584    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w" pod "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" (UID: "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d"). InnerVolumeSpecName "kube-api-access-mrh9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:57.358813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.358791    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls" pod "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" (UID: "69b4eb76-ed3e-4a54-b036-5291f5ee1f4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:57.373242 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.373215    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" path="/var/lib/kubelet/pods/d5237e1b-4892-48aa-8b09-d06e90fbd505/volumes"
Nov 02 08:36:57.458125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.458089    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:57.458125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.458127    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrh9w\" (UniqueName: \"kubernetes.io/projected/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-kube-api-access-mrh9w\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:57.458426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.458141    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:36:57.766990 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.766953    4515 generic.go:356] "Generic (PLEG): container finished" podID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" containerID="4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4" exitCode=0
Nov 02 08:36:57.767203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.767001    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"
Nov 02 08:36:57.767203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.767041    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" event={"ID":"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d","Type":"ContainerDied","Data":"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"}
Nov 02 08:36:57.767203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.767117    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd" event={"ID":"69b4eb76-ed3e-4a54-b036-5291f5ee1f4d","Type":"ContainerDied","Data":"a6834d8aa215ffaa4930298a8a8fe1ad28605d97336d4839c07d7a6a0aac6b10"}
Nov 02 08:36:57.767203 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.767148    4515 scope.go:122] "RemoveContainer" containerID="4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"
Nov 02 08:36:57.776451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.776431    4515 scope.go:122] "RemoveContainer" containerID="4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"
Nov 02 08:36:57.776752 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:36:57.776731    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4\": container with ID starting with 4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4 not found: ID does not exist" containerID="4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"
Nov 02 08:36:57.776849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.776760    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4"} err="failed to get container status \"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4\": rpc error: code = NotFound desc = could not find container \"4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4\": container with ID starting with 4c9514c341be7b1205f5e410fa9b708b192d83d95ace497597dd36fe75e6fcb4 not found: ID does not exist"
Nov 02 08:36:57.819802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.819765    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 08:36:57.823476 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.823455    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9877ce-6c74d4688-k8mqd"]
Nov 02 08:36:57.872713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:57.872672    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:58.451059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:58.451014    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:58.451491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:58.451018    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:59.372508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:59.372474    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" path="/var/lib/kubelet/pods/69b4eb76-ed3e-4a54-b036-5291f5ee1f4d/volumes"
Nov 02 08:36:59.975697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:36:59.975659    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:37:00.009261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:00.009220    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:37:03.082954 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:03.082909    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 08:37:03.115604 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:03.115570    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 08:37:04.976793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:04.976750    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing" containerID="cri-o://67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2" gracePeriod=25
Nov 02 08:37:05.010596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.010528    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing" containerID="cri-o://0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2" gracePeriod=25
Nov 02 08:37:05.789888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.789822    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 08:37:05.814906 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.814871    4515 generic.go:356] "Generic (PLEG): container finished" podID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerID="27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81" exitCode=0
Nov 02 08:37:05.815104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.814925    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"
Nov 02 08:37:05.815104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.814959    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" event={"ID":"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80","Type":"ContainerDied","Data":"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"}
Nov 02 08:37:05.815104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.815016    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc" event={"ID":"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80","Type":"ContainerDied","Data":"51da1f02bfa683a8c5a6221865553eacb5a9fc0b0aac4712a1f15ef7830e8f5c"}
Nov 02 08:37:05.815104 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.815041    4515 scope.go:122] "RemoveContainer" containerID="27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"
Nov 02 08:37:05.824655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.824635    4515 scope.go:122] "RemoveContainer" containerID="27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"
Nov 02 08:37:05.824962 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:05.824935    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81\": container with ID starting with 27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81 not found: ID does not exist" containerID="27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"
Nov 02 08:37:05.825029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.824976    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81"} err="failed to get container status \"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81\": rpc error: code = NotFound desc = could not find container \"27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81\": container with ID starting with 27ef8476e641b10746f7ce808ed230ce22a6b1dbcdb47fec5fd38a174f807e81 not found: ID does not exist"
Nov 02 08:37:05.847289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.847200    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\" (UniqueName: \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\") pod \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") "
Nov 02 08:37:05.847521 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.847373    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\") pod \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\" (UID: \"1d1e377b-e1e2-4394-ace7-3f72e1a1ce80\") "
Nov 02 08:37:05.847837 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.847811    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume" pod "1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" (UID: "1d1e377b-e1e2-4394-ace7-3f72e1a1ce80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:05.848849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.848822    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:05.849992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.849896    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5" pod "1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" (UID: "1d1e377b-e1e2-4394-ace7-3f72e1a1ce80"). InnerVolumeSpecName "kube-api-access-6b7b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:05.949883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:05.949831    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7b5\" (UniqueName: \"kubernetes.io/projected/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80-kube-api-access-6b7b5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:06.161508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:06.161474    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 08:37:06.165443 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:06.165416    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9877ce-6fc6c4dccf-wmxzc"]
Nov 02 08:37:07.372627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:07.372593    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" path="/var/lib/kubelet/pods/1d1e377b-e1e2-4394-ace7-3f72e1a1ce80/volumes"
Nov 02 08:37:08.083599 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.083537    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" podUID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" containerName="payload-processing" containerID="cri-o://fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d" gracePeriod=25
Nov 02 08:37:08.116000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.115958    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" podUID="b7735656-137d-412b-8e56-4efb71a4f697" containerName="payload-pre-processing" containerID="cri-o://2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a" gracePeriod=25
Nov 02 08:37:08.376224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.376197    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 08:37:08.400625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.400601    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 08:37:08.460097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.451445    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:08.460097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.451601    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:08.475640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.475587    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:37:08.476073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.475847    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy" containerID="cri-o://3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887" gracePeriod=30
Nov 02 08:37:08.481519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.481490    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\" (UniqueName: \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\") pod \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") "
Nov 02 08:37:08.481669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.481564    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\") pod \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\" (UID: \"bbf7fff7-9562-4547-b7b9-735f3c2503cd\") "
Nov 02 08:37:08.481669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.481655    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\" (UniqueName: \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\") pod \"b7735656-137d-412b-8e56-4efb71a4f697\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") "
Nov 02 08:37:08.481788 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.481691    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\") pod \"b7735656-137d-412b-8e56-4efb71a4f697\" (UID: \"b7735656-137d-412b-8e56-4efb71a4f697\") "
Nov 02 08:37:08.482522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.482495    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume" pod "b7735656-137d-412b-8e56-4efb71a4f697" (UID: "b7735656-137d-412b-8e56-4efb71a4f697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:08.483302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.483272    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume" pod "bbf7fff7-9562-4547-b7b9-735f3c2503cd" (UID: "bbf7fff7-9562-4547-b7b9-735f3c2503cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:08.484852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.484825    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv" pod "bbf7fff7-9562-4547-b7b9-735f3c2503cd" (UID: "bbf7fff7-9562-4547-b7b9-735f3c2503cd"). InnerVolumeSpecName "kube-api-access-j25fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:08.485216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.485187    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9" pod "b7735656-137d-412b-8e56-4efb71a4f697" (UID: "b7735656-137d-412b-8e56-4efb71a4f697"). InnerVolumeSpecName "kube-api-access-m8gk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:08.582773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.582724    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j25fv\" (UniqueName: \"kubernetes.io/projected/bbf7fff7-9562-4547-b7b9-735f3c2503cd-kube-api-access-j25fv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.582773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.582754    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf7fff7-9562-4547-b7b9-735f3c2503cd-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.582773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.582770    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8gk9\" (UniqueName: \"kubernetes.io/projected/b7735656-137d-412b-8e56-4efb71a4f697-kube-api-access-m8gk9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.583232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.582783    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7735656-137d-412b-8e56-4efb71a4f697-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.838266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.838222    4515 generic.go:356] "Generic (PLEG): container finished" podID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" containerID="fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d" exitCode=0
Nov 02 08:37:08.838483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.838274    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"
Nov 02 08:37:08.838483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.838303    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" event={"ID":"bbf7fff7-9562-4547-b7b9-735f3c2503cd","Type":"ContainerDied","Data":"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"}
Nov 02 08:37:08.838483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.838389    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm" event={"ID":"bbf7fff7-9562-4547-b7b9-735f3c2503cd","Type":"ContainerDied","Data":"176fd895985b4d0eaff4a64fad9278eecc23c420c45ba45f37db5ef6ea50737d"}
Nov 02 08:37:08.838483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.838412    4515 scope.go:122] "RemoveContainer" containerID="fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"
Nov 02 08:37:08.839793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.839770    4515 generic.go:356] "Generic (PLEG): container finished" podID="b7735656-137d-412b-8e56-4efb71a4f697" containerID="2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a" exitCode=0
Nov 02 08:37:08.839889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.839811    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" event={"ID":"b7735656-137d-412b-8e56-4efb71a4f697","Type":"ContainerDied","Data":"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"}
Nov 02 08:37:08.839889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.839827    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"
Nov 02 08:37:08.839889 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.839850    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f" event={"ID":"b7735656-137d-412b-8e56-4efb71a4f697","Type":"ContainerDied","Data":"90fb538e53d31e67d1ae17f21cb87b3c3c59c6eaafa529745f8fe86ab398a025"}
Nov 02 08:37:08.849351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.849331    4515 scope.go:122] "RemoveContainer" containerID="fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"
Nov 02 08:37:08.849652 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:08.849629    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d\": container with ID starting with fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d not found: ID does not exist" containerID="fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"
Nov 02 08:37:08.849736 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.849659    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d"} err="failed to get container status \"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d\": rpc error: code = NotFound desc = could not find container \"fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d\": container with ID starting with fda57a5569ac70380a304cd4064d0561e738b9178d56094dd746c45e4322883d not found: ID does not exist"
Nov 02 08:37:08.849736 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.849676    4515 scope.go:122] "RemoveContainer" containerID="2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"
Nov 02 08:37:08.858223 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.858205    4515 scope.go:122] "RemoveContainer" containerID="2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"
Nov 02 08:37:08.858495 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:08.858473    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a\": container with ID starting with 2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a not found: ID does not exist" containerID="2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"
Nov 02 08:37:08.858549 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.858502    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a"} err="failed to get container status \"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a\": rpc error: code = NotFound desc = could not find container \"2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a\": container with ID starting with 2be968b667735a02cda69ada8f1ce35564456961e538d8d65aa2e3115e17462a not found: ID does not exist"
Nov 02 08:37:08.892108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.892081    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 08:37:08.895575 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.895551    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-354660-58c847bbf7-hgzfm"]
Nov 02 08:37:08.930805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.930773    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 08:37:08.934701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:08.934676    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-354660-59855fc979-s2v7f"]
Nov 02 08:37:09.372933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:09.372853    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7735656-137d-412b-8e56-4efb71a4f697" path="/var/lib/kubelet/pods/b7735656-137d-412b-8e56-4efb71a4f697/volumes"
Nov 02 08:37:09.373286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:09.373272    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" path="/var/lib/kubelet/pods/bbf7fff7-9562-4547-b7b9-735f3c2503cd/volumes"
Nov 02 08:37:10.350898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.350869    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:37:10.386850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.386825    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:37:10.505710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.505673    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\") pod \"498e4915-9ad3-441d-95fb-21ea9534d23d\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") "
Nov 02 08:37:10.505898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.505733    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\") pod \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") "
Nov 02 08:37:10.505898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.505791    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\" (UniqueName: \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\") pod \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\" (UID: \"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a\") "
Nov 02 08:37:10.505898 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.505823    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\" (UniqueName: \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\") pod \"498e4915-9ad3-441d-95fb-21ea9534d23d\" (UID: \"498e4915-9ad3-441d-95fb-21ea9534d23d\") "
Nov 02 08:37:10.506186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.506132    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume" pod "498e4915-9ad3-441d-95fb-21ea9534d23d" (UID: "498e4915-9ad3-441d-95fb-21ea9534d23d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:10.506256 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.506206    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume" pod "f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" (UID: "f0b9c9c6-e90b-4b5c-ac17-3322df328f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:10.506341 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.506293    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498e4915-9ad3-441d-95fb-21ea9534d23d-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:10.507957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.507938    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6" pod "498e4915-9ad3-441d-95fb-21ea9534d23d" (UID: "498e4915-9ad3-441d-95fb-21ea9534d23d"). InnerVolumeSpecName "kube-api-access-7w5c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:10.508029 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.507974    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9" pod "f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" (UID: "f0b9c9c6-e90b-4b5c-ac17-3322df328f6a"). InnerVolumeSpecName "kube-api-access-sfkv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:10.607044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.606956    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:10.607044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.606989    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfkv9\" (UniqueName: \"kubernetes.io/projected/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a-kube-api-access-sfkv9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:10.607044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.607002    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5c6\" (UniqueName: \"kubernetes.io/projected/498e4915-9ad3-441d-95fb-21ea9534d23d-kube-api-access-7w5c6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:10.668845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.668803    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:37:10.853371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.853327    4515 generic.go:356] "Generic (PLEG): container finished" podID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerID="bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862" exitCode=0
Nov 02 08:37:10.853371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.853365    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"
Nov 02 08:37:10.853650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.853401    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" event={"ID":"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a","Type":"ContainerDied","Data":"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"}
Nov 02 08:37:10.853650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.853460    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w" event={"ID":"f0b9c9c6-e90b-4b5c-ac17-3322df328f6a","Type":"ContainerDied","Data":"031ff2e6cb8bc1c28837dbd20ecfb751acf23f889f7b1c8927609560b0748e74"}
Nov 02 08:37:10.853650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.853486    4515 scope.go:122] "RemoveContainer" containerID="bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"
Nov 02 08:37:10.854975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.854952    4515 generic.go:356] "Generic (PLEG): container finished" podID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerID="0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817" exitCode=0
Nov 02 08:37:10.855106 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.855001    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"
Nov 02 08:37:10.855106 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.855043    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" event={"ID":"498e4915-9ad3-441d-95fb-21ea9534d23d","Type":"ContainerDied","Data":"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"}
Nov 02 08:37:10.855106 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.855085    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx" event={"ID":"498e4915-9ad3-441d-95fb-21ea9534d23d","Type":"ContainerDied","Data":"00f181efb1ef3121df288350bf751a3daea43e94c173483f4bc3898e31cf64a4"}
Nov 02 08:37:10.862865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.862844    4515 scope.go:122] "RemoveContainer" containerID="bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"
Nov 02 08:37:10.863192 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:10.863173    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862\": container with ID starting with bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862 not found: ID does not exist" containerID="bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"
Nov 02 08:37:10.863289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.863203    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862"} err="failed to get container status \"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862\": rpc error: code = NotFound desc = could not find container \"bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862\": container with ID starting with bdb2476133d2efb6e8c0a177786da14adaa53d78b56a630981710d64479ee862 not found: ID does not exist"
Nov 02 08:37:10.863289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.863222    4515 scope.go:122] "RemoveContainer" containerID="0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"
Nov 02 08:37:10.872259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.872237    4515 scope.go:122] "RemoveContainer" containerID="0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"
Nov 02 08:37:10.872566 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:10.872546    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817\": container with ID starting with 0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817 not found: ID does not exist" containerID="0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"
Nov 02 08:37:10.872618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.872577    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817"} err="failed to get container status \"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817\": rpc error: code = NotFound desc = could not find container \"0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817\": container with ID starting with 0a6c4a13d8b44cfd53dcbe8206295c271cd00f73b5ec309d002f497223f80817 not found: ID does not exist"
Nov 02 08:37:10.925201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.925162    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:37:10.929799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.929770    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3d8293-67c9869b9f-fkbsx"]
Nov 02 08:37:10.969726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.969689    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:37:10.975677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:10.975644    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d8293-57c89ff647-q8t6w"]
Nov 02 08:37:11.372296 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:11.372258    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" path="/var/lib/kubelet/pods/498e4915-9ad3-441d-95fb-21ea9534d23d/volumes"
Nov 02 08:37:11.372719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:11.372709    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" path="/var/lib/kubelet/pods/f0b9c9c6-e90b-4b5c-ac17-3322df328f6a/volumes"
Nov 02 08:37:12.012773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.012735    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" containerID="cri-o://5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25" gracePeriod=45
Nov 02 08:37:12.092113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.092073    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused"
Nov 02 08:37:12.272328 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.272253    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 08:37:12.426015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.425974    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426047    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426095    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426148    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\" (UniqueName: \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426192    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426259    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\" (UniqueName: \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426383    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\") pod \"243fd671-4009-4c9c-a604-3c557af7aa03\" (UID: \"243fd671-4009-4c9c-a604-3c557af7aa03\") "
Nov 02 08:37:12.426539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426474    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.426939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426633    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.426939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426759    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.426939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426779    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.426939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.426790    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.428896 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.428859    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.429173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.429148    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:12.429173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.429162    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q" pod "243fd671-4009-4c9c-a604-3c557af7aa03" (UID: "243fd671-4009-4c9c-a604-3c557af7aa03"). InnerVolumeSpecName "kube-api-access-kc79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:12.528182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.528102    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243fd671-4009-4c9c-a604-3c557af7aa03-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.528182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.528131    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc79q\" (UniqueName: \"kubernetes.io/projected/243fd671-4009-4c9c-a604-3c557af7aa03-kube-api-access-kc79q\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.528182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.528141    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243fd671-4009-4c9c-a604-3c557af7aa03-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.868261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.868169    4515 generic.go:356] "Generic (PLEG): container finished" podID="243fd671-4009-4c9c-a604-3c557af7aa03" containerID="5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25" exitCode=0
Nov 02 08:37:12.868261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.868217    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"
Nov 02 08:37:12.868542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.868254    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" event={"ID":"243fd671-4009-4c9c-a604-3c557af7aa03","Type":"ContainerDied","Data":"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"}
Nov 02 08:37:12.868542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.868322    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78" event={"ID":"243fd671-4009-4c9c-a604-3c557af7aa03","Type":"ContainerDied","Data":"969008e92aba0dd5d06176cb9985227652a7e3a1eeef6b6777568ce1fd75bef5"}
Nov 02 08:37:12.868542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.868345    4515 scope.go:122] "RemoveContainer" containerID="5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"
Nov 02 08:37:12.877718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.877700    4515 scope.go:122] "RemoveContainer" containerID="5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"
Nov 02 08:37:12.878076 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:12.878044    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25\": container with ID starting with 5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25 not found: ID does not exist" containerID="5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"
Nov 02 08:37:12.878172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.878083    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25"} err="failed to get container status \"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25\": rpc error: code = NotFound desc = could not find container \"5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25\": container with ID starting with 5660677f63c3b6ebb25c5d044fc3f21a4c6c1e31009460387bfbaa659da56c25 not found: ID does not exist"
Nov 02 08:37:12.925853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.925821    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 08:37:12.938066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:12.938041    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-354660/premium-w3-354660-kserve-79dc9d56cf-xwt78"]
Nov 02 08:37:13.004956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.004915    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:13.005112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.004916    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:13.215289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.215203    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 08:37:13.244809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.244776    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 08:37:13.373466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.373433    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" path="/var/lib/kubelet/pods/243fd671-4009-4c9c-a604-3c557af7aa03/volumes"
Nov 02 08:37:13.743038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.743009    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:37:13.840113 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840078    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840120    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840248    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840292    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840356    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840389    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840463    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840483    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840514    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\") pod \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\" (UID: \"ce9b6f41-306e-42e9-bcf8-9fc9985507cb\") "
Nov 02 08:37:13.840749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840545    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:13.840749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840674    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:13.840920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840794    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:13.840920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840857    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:13.840920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840896    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:13.841115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.840996    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.841115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.841013    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.841115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.841026    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.841115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.841039    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.841115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.841053    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.842532 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.842501    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:13.842675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.842614    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:13.842675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.842660    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:13.842801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.842784    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn" pod "ce9b6f41-306e-42e9-bcf8-9fc9985507cb" (UID: "ce9b6f41-306e-42e9-bcf8-9fc9985507cb"). InnerVolumeSpecName "kube-api-access-dw7jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:13.875868 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.875833    4515 generic.go:356] "Generic (PLEG): container finished" podID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerID="3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887" exitCode=0
Nov 02 08:37:13.876066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.875884    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"
Nov 02 08:37:13.876066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.875918    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerDied","Data":"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"}
Nov 02 08:37:13.876066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.875980    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n" event={"ID":"ce9b6f41-306e-42e9-bcf8-9fc9985507cb","Type":"ContainerDied","Data":"69b38a07751211885993f5b96d98cabf0d53437fa8b7424f79aaf0dc9fdd7fc5"}
Nov 02 08:37:13.876066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.876010    4515 scope.go:122] "RemoveContainer" containerID="3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"
Nov 02 08:37:13.889190 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.889163    4515 scope.go:122] "RemoveContainer" containerID="77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"
Nov 02 08:37:13.899605 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.899580    4515 scope.go:122] "RemoveContainer" containerID="3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"
Nov 02 08:37:13.899960 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:13.899934    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887\": container with ID starting with 3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887 not found: ID does not exist" containerID="3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"
Nov 02 08:37:13.900076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.899966    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887"} err="failed to get container status \"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887\": rpc error: code = NotFound desc = could not find container \"3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887\": container with ID starting with 3634fa7de734268b07e939e293681754cf70e1f7ffedf71bcd2bcb0ff9526887 not found: ID does not exist"
Nov 02 08:37:13.900076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.899985    4515 scope.go:122] "RemoveContainer" containerID="77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"
Nov 02 08:37:13.900289 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:13.900262    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930\": container with ID starting with 77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930 not found: ID does not exist" containerID="77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"
Nov 02 08:37:13.900381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.900296    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930"} err="failed to get container status \"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930\": rpc error: code = NotFound desc = could not find container \"77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930\": container with ID starting with 77d5a572355016d18e79f4903cb5ffc47656227b56c5547c673bc5c963d32930 not found: ID does not exist"
Nov 02 08:37:13.929325 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.929279    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:37:13.938842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.938812    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a44352-openshift-default-5955f6c46c-tss8n"]
Nov 02 08:37:13.942365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.942345    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.942422 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.942368    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.942422 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.942378    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw7jn\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-kube-api-access-dw7jn\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.942422 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:13.942388    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b6f41-306e-42e9-bcf8-9fc9985507cb-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:14.547719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:14.547684    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:37:15.037555 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.037521    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 08:37:15.038446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.038392    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" podUID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" containerName="maas-api" containerID="cri-o://16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566" gracePeriod=30
Nov 02 08:37:15.295802 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.295729    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 08:37:15.374294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.374261    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" path="/var/lib/kubelet/pods/ce9b6f41-306e-42e9-bcf8-9fc9985507cb/volumes"
Nov 02 08:37:15.457922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.457864    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\") pod \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") "
Nov 02 08:37:15.458128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.457961    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\" (UniqueName: \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\") pod \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") "
Nov 02 08:37:15.458128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.458013    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\") pod \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") "
Nov 02 08:37:15.458128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.458081    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\") pod \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\" (UID: \"638f4e75-80bb-4d6c-b813-b5cbeaa1e155\") "
Nov 02 08:37:15.458450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.458420    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca" pod "638f4e75-80bb-4d6c-b813-b5cbeaa1e155" (UID: "638f4e75-80bb-4d6c-b813-b5cbeaa1e155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:15.460370 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.460339    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs" pod "638f4e75-80bb-4d6c-b813-b5cbeaa1e155" (UID: "638f4e75-80bb-4d6c-b813-b5cbeaa1e155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:15.460649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.460620    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls" pod "638f4e75-80bb-4d6c-b813-b5cbeaa1e155" (UID: "638f4e75-80bb-4d6c-b813-b5cbeaa1e155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:15.460782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.460755    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58" pod "638f4e75-80bb-4d6c-b813-b5cbeaa1e155" (UID: "638f4e75-80bb-4d6c-b813-b5cbeaa1e155"). InnerVolumeSpecName "kube-api-access-ztr58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:15.559165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.559087    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:15.559165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.559114    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztr58\" (UniqueName: \"kubernetes.io/projected/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-kube-api-access-ztr58\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:15.559165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.559126    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:15.559165 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.559135    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638f4e75-80bb-4d6c-b813-b5cbeaa1e155-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:15.890601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.890509    4515 generic.go:356] "Generic (PLEG): container finished" podID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" containerID="16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566" exitCode=0
Nov 02 08:37:15.890601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.890564    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" event={"ID":"638f4e75-80bb-4d6c-b813-b5cbeaa1e155","Type":"ContainerDied","Data":"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"}
Nov 02 08:37:15.890601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.890569    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"
Nov 02 08:37:15.890873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.890643    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8" event={"ID":"638f4e75-80bb-4d6c-b813-b5cbeaa1e155","Type":"ContainerDied","Data":"1999f818bb584cac63bf70365c15c315dc94251b6b4847b1b425e74054b8f9f1"}
Nov 02 08:37:15.890873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.890665    4515 scope.go:122] "RemoveContainer" containerID="16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"
Nov 02 08:37:15.900250 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.900228    4515 scope.go:122] "RemoveContainer" containerID="16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"
Nov 02 08:37:15.900581 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:15.900561    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566\": container with ID starting with 16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566 not found: ID does not exist" containerID="16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"
Nov 02 08:37:15.900655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.900589    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566"} err="failed to get container status \"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566\": rpc error: code = NotFound desc = could not find container \"16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566\": container with ID starting with 16b54fb9d23cca3f7fc5560dfe47441502d1481cd14ed5e1d410a90cd0bf3566 not found: ID does not exist"
Nov 02 08:37:15.944030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.943995    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 08:37:15.947944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:15.947922    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7f19e8-68cd74d8b8-zjxv8"]
Nov 02 08:37:17.372040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:17.372006    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" path="/var/lib/kubelet/pods/638f4e75-80bb-4d6c-b813-b5cbeaa1e155/volumes"
Nov 02 08:37:17.722434 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:17.722355    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:37:18.216156 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:18.216117    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing" containerID="cri-o://290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57" gracePeriod=25
Nov 02 08:37:18.245620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:18.245581    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing" containerID="cri-o://e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c" gracePeriod=25
Nov 02 08:37:18.710770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:18.710725    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:18.711189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:18.710786    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:19.548750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:19.548699    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing" containerID="cri-o://70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846" gracePeriod=25
Nov 02 08:37:21.229276 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:21.229233    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:21.266483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:21.266440    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 08:37:21.266739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:21.266698    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy" containerID="cri-o://ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a" gracePeriod=30
Nov 02 08:37:23.004706 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.004655    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:23.005117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.004748    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:23.783855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.783822    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:37:23.784105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.784079    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" containerID="cri-o://83519c6bdcbf45817cd4091cb77a010b29ab9af27eb3c5b7e008acc413d1423c" gracePeriod=30
Nov 02 08:37:23.897484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.897445    4515 patch_prober.go:29] interesting pod/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:37:23.897668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:23.897514    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:37:24.272249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272212    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:24.272675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272665    4515 cpu_manager.go:414] "containerMap: removing container" podUID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" containerName="maas-api"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272676    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" containerName="maas-api"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272689    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272694    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272705    4515 cpu_manager.go:414] "containerMap: removing container" podUID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" containerName="payload-processing"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272710    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" containerName="payload-processing"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272716    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272721    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:37:24.272728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272726    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272732    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272742    4515 cpu_manager.go:414] "containerMap: removing container" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272747    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272753    4515 cpu_manager.go:414] "containerMap: removing container" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272757    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272764    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272768    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272774    4515 cpu_manager.go:414] "containerMap: removing container" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272778    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272784    4515 cpu_manager.go:414] "containerMap: removing container" podUID="0d6b8eda-0315-4668-abed-16075315bd17" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272788    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6b8eda-0315-4668-abed-16075315bd17" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272797    4515 cpu_manager.go:414] "containerMap: removing container" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272802    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272808    4515 cpu_manager.go:414] "containerMap: removing container" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272813    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272821    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272826    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272836    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272841    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272846    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272851    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272857    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272864    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272870    4515 cpu_manager.go:414] "containerMap: removing container" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" containerName="test-curl-7361-8b5248"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272875    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" containerName="test-curl-7361-8b5248"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272883    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" containerName="maas-api"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272887    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" containerName="maas-api"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272895    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b7735656-137d-412b-8e56-4efb71a4f697" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272900    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7735656-137d-412b-8e56-4efb71a4f697" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272908    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272912    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272920    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e54c8c31-e110-431c-8233-7e7222e45ddf" containerName="maas-api"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272925    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54c8c31-e110-431c-8233-7e7222e45ddf" containerName="maas-api"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272931    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272935    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272945    4515 cpu_manager.go:414] "containerMap: removing container" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272949    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272957    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272961    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272967    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272971    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272979    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8d045dba-a7af-4ff3-988c-332d14f13ce2" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272983    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d045dba-a7af-4ff3-988c-332d14f13ce2" containerName="main"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272990    4515 cpu_manager.go:414] "containerMap: removing container" podUID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" containerName="maas-api"
Nov 02 08:37:24.272991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.272995    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" containerName="maas-api"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273050    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a79ad55-c933-453f-8924-254ec0d0011e" containerName="payload-pre-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273059    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273064    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea689304-59b3-46c4-9be7-d12d25a78383" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273072    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d045dba-a7af-4ff3-988c-332d14f13ce2" containerName="main"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273077    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b4eb76-ed3e-4a54-b036-5291f5ee1f4d" containerName="maas-api"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273083    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="638f4e75-80bb-4d6c-b813-b5cbeaa1e155" containerName="maas-api"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273088    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd38826-5176-4d0c-ba84-d0797d6ed04c" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273094    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf7fff7-9562-4547-b7b9-735f3c2503cd" containerName="payload-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273099    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273104    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed078af5-0ce7-4c07-9e6f-d4ca95a878d1" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273119    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf1506e-87b5-4951-a2fc-64b71d997a0b" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273124    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f35d01-275a-4261-a631-8d0e008f8760" containerName="payload-pre-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273129    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed74f42a-49c1-4d00-be11-40e7d9656da4" containerName="payload-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273133    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="498e4915-9ad3-441d-95fb-21ea9534d23d" containerName="payload-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273138    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6b8eda-0315-4668-abed-16075315bd17" containerName="main"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273144    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="243fd671-4009-4c9c-a604-3c557af7aa03" containerName="main"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273149    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1e377b-e1e2-4394-ace7-3f72e1a1ce80" containerName="payload-pre-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273155    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b9c9c6-e90b-4b5c-ac17-3322df328f6a" containerName="payload-pre-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273163    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273168    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35ac2e0-4ee8-4752-9a1f-51229b40e979" containerName="payload-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273173    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4d2d48-36c3-4144-9b19-a60aa603d91d" containerName="payload-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273177    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf02058e-4a50-4aef-8dd4-3d6bb6497ff2" containerName="test-curl-7361-8b5248"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273183    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7be5b5-9c26-43fd-9026-06dcc11ce70f" containerName="maas-api"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273188    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7735656-137d-412b-8e56-4efb71a4f697" containerName="payload-pre-processing"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273193    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273198    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54c8c31-e110-431c-8233-7e7222e45ddf" containerName="maas-api"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273203    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9b6f41-306e-42e9-bcf8-9fc9985507cb" containerName="istio-proxy"
Nov 02 08:37:24.274539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.273209    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="117b6b36-a1e8-4369-9ae9-e4d2f7299419" containerName="main"
Nov 02 08:37:24.278294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.278277    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.301259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.301228    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:24.320713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.320678    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-354660-openshift-default-dockercfg-kgbn5\""
Nov 02 08:37:24.359883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.359848    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360044 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.359921    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360063    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360108    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360136    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360218    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360252    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360277    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8t9\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.360393 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.360364    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461352    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461398    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8t9\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461416    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461541    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461643    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461681    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461725    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.461810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461773    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461848    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461884    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461902    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462043 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.461992    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.462241    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.462440 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.462417    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.463744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.463725    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.464095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.464079    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.486576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.486548    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.486843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.486820    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8t9\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d9669\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.590942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.590863    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:24.911553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.911518    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:24.912748 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:37:24.912724    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e20aaa7_41d3_4549_ac18_73e616298ca9.slice/crio-6815bc7624c358b251e3d033337f61f223a3312b63908780a668eafaee729fc9 WatchSource:0}: Error finding container 6815bc7624c358b251e3d033337f61f223a3312b63908780a668eafaee729fc9: Status 404 returned error can't find the container with id 6815bc7624c358b251e3d033337f61f223a3312b63908780a668eafaee729fc9
Nov 02 08:37:24.914997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.914965    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:37:24.915076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.915031    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:37:24.915076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.915060    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:37:24.940486 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:24.940459    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" event={"ID":"3e20aaa7-41d3-4549-ac18-73e616298ca9","Type":"ContainerStarted","Data":"6815bc7624c358b251e3d033337f61f223a3312b63908780a668eafaee729fc9"}
Nov 02 08:37:25.225913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.225880    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:37:25.270727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.270692    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\") pod \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") "
Nov 02 08:37:25.270908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.270761    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\" (UniqueName: \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\") pod \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\" (UID: \"52099437-e3e1-4bd8-a9b7-be59c98e14a6\") "
Nov 02 08:37:25.272355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.271880    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume" pod "52099437-e3e1-4bd8-a9b7-be59c98e14a6" (UID: "52099437-e3e1-4bd8-a9b7-be59c98e14a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:25.273875 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.273846    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx" pod "52099437-e3e1-4bd8-a9b7-be59c98e14a6" (UID: "52099437-e3e1-4bd8-a9b7-be59c98e14a6"). InnerVolumeSpecName "kube-api-access-vjrtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:25.278281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.278257    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:37:25.372426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.372390    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\" (UniqueName: \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\") pod \"110fce66-0aad-4dee-be59-8e66fd14468d\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") "
Nov 02 08:37:25.372601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.372463    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\") pod \"110fce66-0aad-4dee-be59-8e66fd14468d\" (UID: \"110fce66-0aad-4dee-be59-8e66fd14468d\") "
Nov 02 08:37:25.372793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.372776    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52099437-e3e1-4bd8-a9b7-be59c98e14a6-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.372861 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.372798    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrtx\" (UniqueName: \"kubernetes.io/projected/52099437-e3e1-4bd8-a9b7-be59c98e14a6-kube-api-access-vjrtx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.374701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.374666    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume" pod "110fce66-0aad-4dee-be59-8e66fd14468d" (UID: "110fce66-0aad-4dee-be59-8e66fd14468d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:25.377787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.377755    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq" pod "110fce66-0aad-4dee-be59-8e66fd14468d" (UID: "110fce66-0aad-4dee-be59-8e66fd14468d"). InnerVolumeSpecName "kube-api-access-dlcxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:25.474109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.474073    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/110fce66-0aad-4dee-be59-8e66fd14468d-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.474236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.474111    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcxq\" (UniqueName: \"kubernetes.io/projected/110fce66-0aad-4dee-be59-8e66fd14468d-kube-api-access-dlcxq\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.947799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.947759    4515 generic.go:356] "Generic (PLEG): container finished" podID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerID="0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2" exitCode=0
Nov 02 08:37:25.948017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.947814    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"
Nov 02 08:37:25.948017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.947834    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" event={"ID":"52099437-e3e1-4bd8-a9b7-be59c98e14a6","Type":"ContainerDied","Data":"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"}
Nov 02 08:37:25.948017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.947891    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc" event={"ID":"52099437-e3e1-4bd8-a9b7-be59c98e14a6","Type":"ContainerDied","Data":"a8e0d5a25eae185a19f889775ab032b82319b20f6f71cf74613d8a67b585342a"}
Nov 02 08:37:25.948017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.947916    4515 scope.go:122] "RemoveContainer" containerID="0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"
Nov 02 08:37:25.949896 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.949867    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" event={"ID":"3e20aaa7-41d3-4549-ac18-73e616298ca9","Type":"ContainerStarted","Data":"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"}
Nov 02 08:37:25.951569 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.951542    4515 generic.go:356] "Generic (PLEG): container finished" podID="110fce66-0aad-4dee-be59-8e66fd14468d" containerID="67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2" exitCode=0
Nov 02 08:37:25.951685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.951585    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"
Nov 02 08:37:25.951685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.951595    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" event={"ID":"110fce66-0aad-4dee-be59-8e66fd14468d","Type":"ContainerDied","Data":"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"}
Nov 02 08:37:25.951685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.951631    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb" event={"ID":"110fce66-0aad-4dee-be59-8e66fd14468d","Type":"ContainerDied","Data":"d4c7119447489ac6987411e226da936c4208fc233425a47e94cbfd18f55566b0"}
Nov 02 08:37:25.958010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.957993    4515 scope.go:122] "RemoveContainer" containerID="0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"
Nov 02 08:37:25.958272 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:25.958246    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2\": container with ID starting with 0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2 not found: ID does not exist" containerID="0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"
Nov 02 08:37:25.958355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.958278    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2"} err="failed to get container status \"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2\": rpc error: code = NotFound desc = could not find container \"0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2\": container with ID starting with 0de6533e4a6a7028de6ff151596b35108033e7ac1b7f0446db8dbcc326ba22d2 not found: ID does not exist"
Nov 02 08:37:25.958355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.958294    4515 scope.go:122] "RemoveContainer" containerID="67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"
Nov 02 08:37:25.966425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.966341    4515 scope.go:122] "RemoveContainer" containerID="67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"
Nov 02 08:37:25.966744 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:25.966720    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2\": container with ID starting with 67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2 not found: ID does not exist" containerID="67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"
Nov 02 08:37:25.966839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:25.966761    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2"} err="failed to get container status \"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2\": rpc error: code = NotFound desc = could not find container \"67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2\": container with ID starting with 67f35d4903b2fa12e601ca7309f85b05a1876f04f9979bcd6378df4bfad356a2 not found: ID does not exist"
Nov 02 08:37:26.005020 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.004985    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:37:26.031334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.018468    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a44352-67db99b575-h8vcc"]
Nov 02 08:37:26.055847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.055814    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:37:26.058771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.058744    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a44352-79f898dd97-h6vbb"]
Nov 02 08:37:26.115237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.115172    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podStartSLOduration=2.115150981 podStartE2EDuration="2.115150981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:26.112865659 +0000 UTC m=+4390.875302565" watchObservedRunningTime="2026-11-02 08:37:26.115150981 +0000 UTC m=+4390.877587888"
Nov 02 08:37:26.140487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.140447    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:37:26.591199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.591138    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:26.592708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.592683    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:26.592842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.592745    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:26.595858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.595837    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 08:37:26.688115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688085    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.688289 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688176    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.688427 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688391    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:26.688551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688449    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:26.688764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688742    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.688904 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.688891    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.689032 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689019    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.689138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689128    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.689228 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689211    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.689372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689360    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.689523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689360    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:26.689523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689503    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:26.689657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689645    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\") pod \"159a0cec-6643-43c9-9e64-33a38e868887\" (UID: \"159a0cec-6643-43c9-9e64-33a38e868887\") "
Nov 02 08:37:26.690065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.689965    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:26.690407 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690393    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.690484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690476    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.690537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690530    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a0cec-6643-43c9-9e64-33a38e868887-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.690596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690589    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.690663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690656    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.690956 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.690936    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "kube-api-access-qddpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:26.691478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.691456    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:26.691600 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.691578    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:26.691822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.691806    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo" pod "159a0cec-6643-43c9-9e64-33a38e868887" (UID: "159a0cec-6643-43c9-9e64-33a38e868887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:26.791563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.791528    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddpl\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-kube-api-access-qddpl\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.791563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.791559    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a0cec-6643-43c9-9e64-33a38e868887-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.791563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.791570    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a0cec-6643-43c9-9e64-33a38e868887-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.791779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.791578    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a0cec-6643-43c9-9e64-33a38e868887-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.966514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.966423    4515 generic.go:356] "Generic (PLEG): container finished" podID="159a0cec-6643-43c9-9e64-33a38e868887" containerID="ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a" exitCode=0
Nov 02 08:37:26.966514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.966485    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"
Nov 02 08:37:26.966514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.966493    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerDied","Data":"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"}
Nov 02 08:37:26.966741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.966568    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z" event={"ID":"159a0cec-6643-43c9-9e64-33a38e868887","Type":"ContainerDied","Data":"2ce524cede2d2d4023aac62a6a9baf960cb330ffd9c18767683d645ba89e17ec"}
Nov 02 08:37:26.966741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.966582    4515 scope.go:122] "RemoveContainer" containerID="ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"
Nov 02 08:37:26.977120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.977095    4515 scope.go:122] "RemoveContainer" containerID="1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"
Nov 02 08:37:26.985718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.985695    4515 scope.go:122] "RemoveContainer" containerID="ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"
Nov 02 08:37:26.985955 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:26.985935    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a\": container with ID starting with ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a not found: ID does not exist" containerID="ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"
Nov 02 08:37:26.986028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.985963    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a"} err="failed to get container status \"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a\": rpc error: code = NotFound desc = could not find container \"ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a\": container with ID starting with ba1bcea0aa0ff162e4cf866bdb0b0986e333e4b2242b95a03625f8a04730020a not found: ID does not exist"
Nov 02 08:37:26.986028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.985980    4515 scope.go:122] "RemoveContainer" containerID="1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"
Nov 02 08:37:26.986239 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:26.986222    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a\": container with ID starting with 1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a not found: ID does not exist" containerID="1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"
Nov 02 08:37:26.986295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:26.986246    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a"} err="failed to get container status \"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a\": rpc error: code = NotFound desc = could not find container \"1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a\": container with ID starting with 1d721f5961762a640aeba53a9ca3323e927d964c9bdea3e5cb7645a744bede5a not found: ID does not exist"
Nov 02 08:37:27.041346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.040859    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 08:37:27.052206 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.052175    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7f19e8-openshift-default-67fbc85cc-9lz8z"]
Nov 02 08:37:27.372075 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.372041    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" path="/var/lib/kubelet/pods/110fce66-0aad-4dee-be59-8e66fd14468d/volumes"
Nov 02 08:37:27.372584 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.372569    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159a0cec-6643-43c9-9e64-33a38e868887" path="/var/lib/kubelet/pods/159a0cec-6643-43c9-9e64-33a38e868887/volumes"
Nov 02 08:37:27.372973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.372960    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" path="/var/lib/kubelet/pods/52099437-e3e1-4bd8-a9b7-be59c98e14a6/volumes"
Nov 02 08:37:27.592122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.592082    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:27.592551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:27.592138    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:28.592050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.592018    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:28.592237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.592068    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:28.710573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.710534    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:28.710752 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.710549    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:28.983822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.983511    4515 generic.go:356] "Generic (PLEG): container finished" podID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerID="83519c6bdcbf45817cd4091cb77a010b29ab9af27eb3c5b7e008acc413d1423c" exitCode=0
Nov 02 08:37:28.983822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.983613    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerDied","Data":"83519c6bdcbf45817cd4091cb77a010b29ab9af27eb3c5b7e008acc413d1423c"}
Nov 02 08:37:28.983822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:28.983669    4515 scope.go:122] "RemoveContainer" containerID="bcebaf2f4ce7045167bf1ac58c9c85bdf9c63da0a5424e1beca9f71aa1031b12"
Nov 02 08:37:29.080061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.080033    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:37:29.214591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214501    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214557    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214602    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214668    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214701    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214737    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214807    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.214891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214860    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.215240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214895    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\") pod \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\" (UID: \"6d4ed0f8-7aea-458b-b071-fe76cfbbe123\") "
Nov 02 08:37:29.215240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.214925    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:29.215240 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215199    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:29.215449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215340    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.215449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215355    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.215449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215346    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:29.215449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215407    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:29.215671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.215455    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:29.242940 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.242893    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "kube-api-access-79qjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:29.243140 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.243113    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:29.243658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.243414    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:29.243658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.243569    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy" pod "6d4ed0f8-7aea-458b-b071-fe76cfbbe123" (UID: "6d4ed0f8-7aea-458b-b071-fe76cfbbe123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316372    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316410    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316425    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316439    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316452    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316467    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.332347 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.316481    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qjp\" (UniqueName: \"kubernetes.io/projected/6d4ed0f8-7aea-458b-b071-fe76cfbbe123-kube-api-access-79qjp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:29.591972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.591938    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:29.592164 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.591995    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:29.990069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.989988    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"
Nov 02 08:37:29.990069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.990038    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9" event={"ID":"6d4ed0f8-7aea-458b-b071-fe76cfbbe123","Type":"ContainerDied","Data":"85d95c5d99cad914ccb59d19856893fd88ea9742f2a2a9866f50c500188c8c24"}
Nov 02 08:37:29.990528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:29.990084    4515 scope.go:122] "RemoveContainer" containerID="83519c6bdcbf45817cd4091cb77a010b29ab9af27eb3c5b7e008acc413d1423c"
Nov 02 08:37:30.064213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:30.064171    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:37:30.068505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:30.068482    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea9556-openshift-default-767c695d45-zhtt9"]
Nov 02 08:37:30.591593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:30.591559    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:30.591770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:30.591610    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:31.230600 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:31.230565    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:31.372169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:31.372134    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" path="/var/lib/kubelet/pods/6d4ed0f8-7aea-458b-b071-fe76cfbbe123/volumes"
Nov 02 08:37:31.591290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:31.591252    4515 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 08:37:31.591462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:31.591339    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 08:37:32.770502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:32.770467    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:32.771094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:32.771068    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:32.772741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:32.772711    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:38.506727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.506696    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 08:37:38.539640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.539614    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 08:37:38.541070 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541045    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\") pod \"188801de-30fb-4c45-8c84-adb7d9979a00\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") "
Nov 02 08:37:38.541226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541095    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\" (UniqueName: \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\") pod \"939815fc-f120-4acb-a238-3d2740dff156\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") "
Nov 02 08:37:38.541226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541162    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\" (UniqueName: \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\") pod \"188801de-30fb-4c45-8c84-adb7d9979a00\" (UID: \"188801de-30fb-4c45-8c84-adb7d9979a00\") "
Nov 02 08:37:38.541389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541263    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\") pod \"939815fc-f120-4acb-a238-3d2740dff156\" (UID: \"939815fc-f120-4acb-a238-3d2740dff156\") "
Nov 02 08:37:38.541461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541443    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume" pod "188801de-30fb-4c45-8c84-adb7d9979a00" (UID: "188801de-30fb-4c45-8c84-adb7d9979a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:38.541695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541676    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume" pod "939815fc-f120-4acb-a238-3d2740dff156" (UID: "939815fc-f120-4acb-a238-3d2740dff156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:38.541793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541772    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939815fc-f120-4acb-a238-3d2740dff156-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:38.541793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.541789    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188801de-30fb-4c45-8c84-adb7d9979a00-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:38.543230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.543202    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p" pod "188801de-30fb-4c45-8c84-adb7d9979a00" (UID: "188801de-30fb-4c45-8c84-adb7d9979a00"). InnerVolumeSpecName "kube-api-access-9zx5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:38.543351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.543272    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4" pod "939815fc-f120-4acb-a238-3d2740dff156" (UID: "939815fc-f120-4acb-a238-3d2740dff156"). InnerVolumeSpecName "kube-api-access-qgwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:38.642214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.642181    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwt4\" (UniqueName: \"kubernetes.io/projected/939815fc-f120-4acb-a238-3d2740dff156-kube-api-access-qgwt4\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:38.642214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:38.642211    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx5p\" (UniqueName: \"kubernetes.io/projected/188801de-30fb-4c45-8c84-adb7d9979a00-kube-api-access-9zx5p\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:39.042398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.042358    4515 generic.go:356] "Generic (PLEG): container finished" podID="939815fc-f120-4acb-a238-3d2740dff156" containerID="e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c" exitCode=0
Nov 02 08:37:39.042576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.042418    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"
Nov 02 08:37:39.042576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.042439    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" event={"ID":"939815fc-f120-4acb-a238-3d2740dff156","Type":"ContainerDied","Data":"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"}
Nov 02 08:37:39.042576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.042504    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2" event={"ID":"939815fc-f120-4acb-a238-3d2740dff156","Type":"ContainerDied","Data":"e46f94475d4569ac507c2b1422200233f6ea7a6309c9551915ddbc2b27e188b8"}
Nov 02 08:37:39.042576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.042521    4515 scope.go:122] "RemoveContainer" containerID="e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"
Nov 02 08:37:39.044961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.044907    4515 generic.go:356] "Generic (PLEG): container finished" podID="188801de-30fb-4c45-8c84-adb7d9979a00" containerID="290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57" exitCode=0
Nov 02 08:37:39.044961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.044943    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" event={"ID":"188801de-30fb-4c45-8c84-adb7d9979a00","Type":"ContainerDied","Data":"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"}
Nov 02 08:37:39.045174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.045000    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4" event={"ID":"188801de-30fb-4c45-8c84-adb7d9979a00","Type":"ContainerDied","Data":"efd94b40bb4236c5e5e05119675cc932f71c3b30fc4f3cc4028b32b010b8916b"}
Nov 02 08:37:39.045303 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.045277    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"
Nov 02 08:37:39.065434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.062993    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:39.065434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.063337    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy" containerID="cri-o://887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821" gracePeriod=30
Nov 02 08:37:39.065434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.063385    4515 scope.go:122] "RemoveContainer" containerID="e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"
Nov 02 08:37:39.066492 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:39.066039    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c\": container with ID starting with e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c not found: ID does not exist" containerID="e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"
Nov 02 08:37:39.066492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.066083    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c"} err="failed to get container status \"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c\": rpc error: code = NotFound desc = could not find container \"e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c\": container with ID starting with e51226d14c498e32e3dcc1dbfcb2500f2040dad94911df0fd9f780c2322d1a7c not found: ID does not exist"
Nov 02 08:37:39.066492 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.066111    4515 scope.go:122] "RemoveContainer" containerID="290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"
Nov 02 08:37:39.080800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.080775    4515 scope.go:122] "RemoveContainer" containerID="290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"
Nov 02 08:37:39.081147 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:39.081126    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57\": container with ID starting with 290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57 not found: ID does not exist" containerID="290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"
Nov 02 08:37:39.081210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.081156    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57"} err="failed to get container status \"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57\": rpc error: code = NotFound desc = could not find container \"290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57\": container with ID starting with 290c246c7611d0ec2622aa68c412ac1c923f6a6520895a0479a55278403eab57 not found: ID does not exist"
Nov 02 08:37:39.112241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.112209    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 08:37:39.124832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.124805    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7f19e8-8595d55f79-f6dd4"]
Nov 02 08:37:39.207974 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.207935    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 08:37:39.212041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.212011    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7f19e8-548b74dc8c-wrxq2"]
Nov 02 08:37:39.372110 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.372024    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" path="/var/lib/kubelet/pods/188801de-30fb-4c45-8c84-adb7d9979a00/volumes"
Nov 02 08:37:39.372558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.372536    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939815fc-f120-4acb-a238-3d2740dff156" path="/var/lib/kubelet/pods/939815fc-f120-4acb-a238-3d2740dff156/volumes"
Nov 02 08:37:39.806710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.806680    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:37:39.855134 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.855097    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\" (UniqueName: \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\") pod \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") "
Nov 02 08:37:39.855329 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.855243    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\") pod \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\" (UID: \"6f3a7eb8-ec63-4a01-91b6-caa289e632dd\") "
Nov 02 08:37:39.855635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.855613    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume" pod "6f3a7eb8-ec63-4a01-91b6-caa289e632dd" (UID: "6f3a7eb8-ec63-4a01-91b6-caa289e632dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:39.857686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.857666    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd" pod "6f3a7eb8-ec63-4a01-91b6-caa289e632dd" (UID: "6f3a7eb8-ec63-4a01-91b6-caa289e632dd"). InnerVolumeSpecName "kube-api-access-5z7hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:39.956872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.956784    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:39.956872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:39.956816    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7hd\" (UniqueName: \"kubernetes.io/projected/6f3a7eb8-ec63-4a01-91b6-caa289e632dd-kube-api-access-5z7hd\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:40.051755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.051721    4515 generic.go:356] "Generic (PLEG): container finished" podID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerID="70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846" exitCode=0
Nov 02 08:37:40.051975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.051787    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"
Nov 02 08:37:40.051975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.051798    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" event={"ID":"6f3a7eb8-ec63-4a01-91b6-caa289e632dd","Type":"ContainerDied","Data":"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"}
Nov 02 08:37:40.051975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.051868    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk" event={"ID":"6f3a7eb8-ec63-4a01-91b6-caa289e632dd","Type":"ContainerDied","Data":"667c9e76146fa62a305ae2cdc93fe7a6d23c6b73624c6395e9983358ba82fe9c"}
Nov 02 08:37:40.051975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.051906    4515 scope.go:122] "RemoveContainer" containerID="70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"
Nov 02 08:37:40.061288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.061266    4515 scope.go:122] "RemoveContainer" containerID="70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"
Nov 02 08:37:40.061577 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:40.061555    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846\": container with ID starting with 70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846 not found: ID does not exist" containerID="70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"
Nov 02 08:37:40.061654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.061591    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846"} err="failed to get container status \"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846\": rpc error: code = NotFound desc = could not find container \"70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846\": container with ID starting with 70010d591fb1c87c7a6a636aac431426b69560e46ff4ae41fa72a2de57066846 not found: ID does not exist"
Nov 02 08:37:40.106096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.106055    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:37:40.108397 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.108368    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea9556-7df86d994f-dsdxk"]
Nov 02 08:37:40.382243 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.382194    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:37:40.382531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:40.382506    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy" containerID="cri-o://3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5" gracePeriod=30
Nov 02 08:37:41.371819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:41.371783    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" path="/var/lib/kubelet/pods/6f3a7eb8-ec63-4a01-91b6-caa289e632dd/volumes"
Nov 02 08:37:44.321670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.321644    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:44.399961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.399875    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.399961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.399941    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.399988    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400030    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400070    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400105    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400136    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400180    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400209    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\") pod \"3e20aaa7-41d3-4549-ac18-73e616298ca9\" (UID: \"3e20aaa7-41d3-4549-ac18-73e616298ca9\") "
Nov 02 08:37:44.400578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400356    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.400578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400380    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.400578 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400497    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400603    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400704    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400727    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400746    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400761    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.400800 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.400775    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.402500 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.402465    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.402814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.402789    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "kube-api-access-rd8t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:44.403282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.403259    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:44.403504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.403405    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo" pod "3e20aaa7-41d3-4549-ac18-73e616298ca9" (UID: "3e20aaa7-41d3-4549-ac18-73e616298ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:44.502089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.502049    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8t9\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-kube-api-access-rd8t9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.502089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.502084    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e20aaa7-41d3-4549-ac18-73e616298ca9-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.502089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.502097    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.502374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.502108    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.502374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:44.502120    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e20aaa7-41d3-4549-ac18-73e616298ca9-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.079414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.079379    4515 generic.go:356] "Generic (PLEG): container finished" podID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerID="887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821" exitCode=0
Nov 02 08:37:45.079603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.079428    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"
Nov 02 08:37:45.079603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.079457    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" event={"ID":"3e20aaa7-41d3-4549-ac18-73e616298ca9","Type":"ContainerDied","Data":"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"}
Nov 02 08:37:45.079603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.079516    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669" event={"ID":"3e20aaa7-41d3-4549-ac18-73e616298ca9","Type":"ContainerDied","Data":"6815bc7624c358b251e3d033337f61f223a3312b63908780a668eafaee729fc9"}
Nov 02 08:37:45.079603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.079548    4515 scope.go:122] "RemoveContainer" containerID="887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"
Nov 02 08:37:45.089626 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.089600    4515 scope.go:122] "RemoveContainer" containerID="887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"
Nov 02 08:37:45.089964 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:45.089943    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821\": container with ID starting with 887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821 not found: ID does not exist" containerID="887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"
Nov 02 08:37:45.090007 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.089976    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821"} err="failed to get container status \"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821\": rpc error: code = NotFound desc = could not find container \"887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821\": container with ID starting with 887911801b7c4e9699a3707b023e2932caf886c673bd65635a26eaa54ecb9821 not found: ID does not exist"
Nov 02 08:37:45.146252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.146215    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:45.158765 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.158735    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d9669"]
Nov 02 08:37:45.371748 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.371670    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" path="/var/lib/kubelet/pods/3e20aaa7-41d3-4549-ac18-73e616298ca9/volumes"
Nov 02 08:37:45.642054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.642026    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:37:45.715539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715500    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715611    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715650    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715694    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715751 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715745    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715783    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715842    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715875    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.715972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.715929    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\") pod \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\" (UID: \"5d1dffec-838b-4f8c-b690-23fea9d4e87c\") "
Nov 02 08:37:45.719168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.719134    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:45.719900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.719438    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:45.719900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.719619    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:45.722403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.722128    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "kube-api-access-dt6q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:45.722640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.722611    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:45.722793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.722665    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:45.723079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.723053    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:45.725003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.724982    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:45.725122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.725054    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo" pod "5d1dffec-838b-4f8c-b690-23fea9d4e87c" (UID: "5d1dffec-838b-4f8c-b690-23fea9d4e87c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:45.817199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817163    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817216    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6q5\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-kube-api-access-dt6q5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817228    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817238    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817247    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817255    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817262    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817270    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:45.817400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:45.817278    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1dffec-838b-4f8c-b690-23fea9d4e87c-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:46.086966 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.086931    4515 generic.go:356] "Generic (PLEG): container finished" podID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerID="3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5" exitCode=0
Nov 02 08:37:46.087161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.086993    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"
Nov 02 08:37:46.087161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.087017    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerDied","Data":"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"}
Nov 02 08:37:46.087161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.087101    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp" event={"ID":"5d1dffec-838b-4f8c-b690-23fea9d4e87c","Type":"ContainerDied","Data":"444fc7e33d692fec93dff80c0e755c88c9f005baaf75d4a9042c2f919c96086e"}
Nov 02 08:37:46.087161 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.087126    4515 scope.go:122] "RemoveContainer" containerID="3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"
Nov 02 08:37:46.097040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.097018    4515 scope.go:122] "RemoveContainer" containerID="95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"
Nov 02 08:37:46.105267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.105247    4515 scope.go:122] "RemoveContainer" containerID="3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"
Nov 02 08:37:46.105543 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:46.105524    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5\": container with ID starting with 3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5 not found: ID does not exist" containerID="3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"
Nov 02 08:37:46.105627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.105556    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5"} err="failed to get container status \"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5\": rpc error: code = NotFound desc = could not find container \"3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5\": container with ID starting with 3791d730b63147304c0e6c3d2c49ce116f1249fd9ac7963e48821acfd0aa94b5 not found: ID does not exist"
Nov 02 08:37:46.105627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.105581    4515 scope.go:122] "RemoveContainer" containerID="95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"
Nov 02 08:37:46.105847 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:46.105823    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4\": container with ID starting with 95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4 not found: ID does not exist" containerID="95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"
Nov 02 08:37:46.105885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.105854    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4"} err="failed to get container status \"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4\": rpc error: code = NotFound desc = could not find container \"95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4\": container with ID starting with 95d30f9caf6c563030e25f03a3cf3a881b460da7fb6942abb4bb52b1a85339a4 not found: ID does not exist"
Nov 02 08:37:46.139396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.139371    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:37:46.152187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:46.152161    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-71a41c-openshift-default-668c99ff8f-lnczp"]
Nov 02 08:37:47.372836 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:47.372799    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" path="/var/lib/kubelet/pods/5d1dffec-838b-4f8c-b690-23fea9d4e87c/volumes"
Nov 02 08:37:47.722601 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:47.722521    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:37:49.799696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:49.799652    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:37:52.387595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:52.387560    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 08:37:52.420914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:52.420880    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 08:37:53.482776 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.482742    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:37:53.483183 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.482928    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerName="discovery" containerID="cri-o://a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde" gracePeriod=30
Nov 02 08:37:53.742662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.742596    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:37:53.892101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892057    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892137    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892178    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892223    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892289    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892380    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.892591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.892435    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\") pod \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\" (UID: \"effbfbc9-cdb1-4160-a90a-6a4b3936d81f\") "
Nov 02 08:37:53.893158 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.893124    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:53.894738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.894706    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:53.894860 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.894807    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:53.895027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.895001    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:53.895144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.895045    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:53.895144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.895084    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:53.895144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.895120    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k" pod "effbfbc9-cdb1-4160-a90a-6a4b3936d81f" (UID: "effbfbc9-cdb1-4160-a90a-6a4b3936d81f"). InnerVolumeSpecName "kube-api-access-jz22k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993288    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-dns-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993350    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz22k\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-kube-api-access-jz22k\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993364    4515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-csr-ca-configmap\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993378    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993389    4515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-cacerts\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993401    4515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-local-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:53.993415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:53.993413    4515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effbfbc9-cdb1-4160-a90a-6a4b3936d81f-istio-kubeconfig\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:37:54.134936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.134899    4515 generic.go:356] "Generic (PLEG): container finished" podID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerID="a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde" exitCode=0
Nov 02 08:37:54.135145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.134948    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"
Nov 02 08:37:54.135145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.134985    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" event={"ID":"effbfbc9-cdb1-4160-a90a-6a4b3936d81f","Type":"ContainerDied","Data":"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"}
Nov 02 08:37:54.135145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.135047    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2" event={"ID":"effbfbc9-cdb1-4160-a90a-6a4b3936d81f","Type":"ContainerDied","Data":"746f0b55c26d4c2c7c0f87bcb167a3176932f530a295f8994a44e381020fcb45"}
Nov 02 08:37:54.135145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.135071    4515 scope.go:122] "RemoveContainer" containerID="a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"
Nov 02 08:37:54.145674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.145647    4515 scope.go:122] "RemoveContainer" containerID="a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"
Nov 02 08:37:54.145934 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:37:54.145912    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde\": container with ID starting with a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde not found: ID does not exist" containerID="a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"
Nov 02 08:37:54.146005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.145944    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde"} err="failed to get container status \"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde\": rpc error: code = NotFound desc = could not find container \"a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde\": container with ID starting with a69fc0de7d3f28ebbac0e6da889788fedcca920db1a7c8425fc787108f69afde not found: ID does not exist"
Nov 02 08:37:54.186494 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.186461    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:37:54.198996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:54.198965    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmcv2"]
Nov 02 08:37:55.371392 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:55.371355    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" path="/var/lib/kubelet/pods/effbfbc9-cdb1-4160-a90a-6a4b3936d81f/volumes"
Nov 02 08:37:57.388831 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:57.388791    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing" containerID="cri-o://f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351" gracePeriod=25
Nov 02 08:37:57.421207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:37:57.421166    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing" containerID="cri-o://059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28" gracePeriod=25
Nov 02 08:38:01.399744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.399708    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400067    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400079    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400091    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400096    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400103    4515 cpu_manager.go:414] "containerMap: removing container" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerName="discovery"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400108    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerName="discovery"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400119    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400123    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400131    4515 cpu_manager.go:414] "containerMap: removing container" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400136    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400143    4515 cpu_manager.go:414] "containerMap: removing container" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400148    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400153    4515 cpu_manager.go:414] "containerMap: removing container" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400158    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400163    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400167    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400175    4515 cpu_manager.go:414] "containerMap: removing container" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400179    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400194    4515 cpu_manager.go:414] "containerMap: removing container" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400199    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400207    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400211    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400218    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400222    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400230    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400234    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6959f0f6-a227-47c5-a6e0-c445d7fc35e2" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400245    4515 cpu_manager.go:414] "containerMap: removing container" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400251    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400321    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="939815fc-f120-4acb-a238-3d2740dff156" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400330    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3a7eb8-ec63-4a01-91b6-caa289e632dd" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400336    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400342    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400349    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="188801de-30fb-4c45-8c84-adb7d9979a00" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400355    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400361    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="effbfbc9-cdb1-4160-a90a-6a4b3936d81f" containerName="discovery"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400367    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1dffec-838b-4f8c-b690-23fea9d4e87c" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400371    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52099437-e3e1-4bd8-a9b7-be59c98e14a6" containerName="payload-pre-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400376    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400381    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="110fce66-0aad-4dee-be59-8e66fd14468d" containerName="payload-processing"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400385    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e20aaa7-41d3-4549-ac18-73e616298ca9" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400390    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="159a0cec-6643-43c9-9e64-33a38e868887" containerName="istio-proxy"
Nov 02 08:38:01.400421 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.400395    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5237e1b-4892-48aa-8b09-d06e90fbd505" containerName="istio-proxy"
Nov 02 08:38:01.403527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.403508    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.419742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.419715    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:01.420076 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.420064    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.420123 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.420078    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4ed0f8-7aea-458b-b071-fe76cfbbe123" containerName="istio-proxy"
Nov 02 08:38:01.422992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.422969    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.431077 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.431055    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9db802-openshift-default-dockercfg-4p9sf\""
Nov 02 08:38:01.431205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.431143    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 08:38:01.433324 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.433292    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:01.448935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.448899    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:01.568531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568491    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.568531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568534    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.568774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568551    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.568774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568644    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.568774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568702    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.568774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568728    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.568913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568775    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.568913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568817    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.568913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568901    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.569012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568918    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.569012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568934    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.569012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.568952    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xmz\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.569120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569007    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.569120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569070    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.569120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569097    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cml\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.569255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569135    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.569255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569180    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.569255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.569211    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.670734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670649    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.670734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670693    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.670734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670719    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.670734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670736    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670828    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670883    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670915    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670940    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.670971    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xmz\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671003    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671044    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671102 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671077    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cml\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671120    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671148    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671206    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671261    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671290    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671333    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.671581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671410    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671646    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671664    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671207    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671701    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671882    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.672182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.672283    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.671966    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.672574 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.672297    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.673551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.673523    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.673698 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.673598    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.674273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.674255    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.674368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.674342    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.694375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.694346    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.694879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.694857    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xmz\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.695237 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.695213    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.695682 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.695664    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cml\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:01.716998 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.716967    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:01.737261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:01.737233    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:02.035781 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.035755    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:02.036719 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:02.036693    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5ea1e3_e6c6_43fc_92c1_35c17088eadc.slice/crio-60f20b248bfff33c91f8db1cdf2800f932163ddd8d54ee6d8bf5c1b409832949 WatchSource:0}: Error finding container 60f20b248bfff33c91f8db1cdf2800f932163ddd8d54ee6d8bf5c1b409832949: Status 404 returned error can't find the container with id 60f20b248bfff33c91f8db1cdf2800f932163ddd8d54ee6d8bf5c1b409832949
Nov 02 08:38:02.038845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.038812    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.038931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.038879    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.038931 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.038905    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.049934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.049911    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:02.050081 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:02.050027    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56911157_12f1_4944_b4e9_130c59e7eec8.slice/crio-0dfdfe603dffd7293ebc0cb43b2aae19287971d7cf37df6bc634d1126f2dea8f WatchSource:0}: Error finding container 0dfdfe603dffd7293ebc0cb43b2aae19287971d7cf37df6bc634d1126f2dea8f: Status 404 returned error can't find the container with id 0dfdfe603dffd7293ebc0cb43b2aae19287971d7cf37df6bc634d1126f2dea8f
Nov 02 08:38:02.053220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.052734    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.053220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.052812    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.053220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.052851    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.179544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.178820    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" event={"ID":"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc","Type":"ContainerStarted","Data":"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"}
Nov 02 08:38:02.179721 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.179581    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" event={"ID":"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc","Type":"ContainerStarted","Data":"60f20b248bfff33c91f8db1cdf2800f932163ddd8d54ee6d8bf5c1b409832949"}
Nov 02 08:38:02.180785 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.180703    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" event={"ID":"56911157-12f1-4944-b4e9-130c59e7eec8","Type":"ContainerStarted","Data":"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"}
Nov 02 08:38:02.180785 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.180753    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" event={"ID":"56911157-12f1-4944-b4e9-130c59e7eec8","Type":"ContainerStarted","Data":"0dfdfe603dffd7293ebc0cb43b2aae19287971d7cf37df6bc634d1126f2dea8f"}
Nov 02 08:38:02.224871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.224805    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podStartSLOduration=1.224787103 podStartE2EDuration="1.224787103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:02.223496993 +0000 UTC m=+4426.985933902" watchObservedRunningTime="2026-11-02 08:38:02.224787103 +0000 UTC m=+4426.987224014"
Nov 02 08:38:02.269890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.269822    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podStartSLOduration=1.2698030550000001 podStartE2EDuration="1.269803055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:02.269415139 +0000 UTC m=+4427.031852049" watchObservedRunningTime="2026-11-02 08:38:02.269803055 +0000 UTC m=+4427.032239965"
Nov 02 08:38:02.730332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.720411    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:02.746333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:02.739421    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:03.737333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:03.735257    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 02 08:38:03.737333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:03.735371    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 02 08:38:03.753338 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:03.752185    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:38:03.753338 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:03.752280    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 02 08:38:04.720335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.720086    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 02 08:38:04.720335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.720173    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 02 08:38:04.737770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.737729    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:38:04.739086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.739051    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 02 08:38:04.834249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.834210    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 08:38:04.835069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:04.835034    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" podUID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" containerName="maas-api" containerID="cri-o://28e32109839975369eee286d45058deae4fbeca6b4021fc306cd13461d60a2d5" gracePeriod=30
Nov 02 08:38:05.200487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.200441    4515 generic.go:356] "Generic (PLEG): container finished" podID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" containerID="28e32109839975369eee286d45058deae4fbeca6b4021fc306cd13461d60a2d5" exitCode=0
Nov 02 08:38:05.200688 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.200618    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" event={"ID":"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9","Type":"ContainerDied","Data":"28e32109839975369eee286d45058deae4fbeca6b4021fc306cd13461d60a2d5"}
Nov 02 08:38:05.276336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.273519    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 08:38:05.411801 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.411770    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\") pod \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") "
Nov 02 08:38:05.411959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.411818    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") pod \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") "
Nov 02 08:38:05.411959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.411897    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\" (UniqueName: \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\") pod \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") "
Nov 02 08:38:05.411959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.411935    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\") pod \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\" (UID: \"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9\") "
Nov 02 08:38:05.412489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.412462    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca" pod "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" (UID: "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:05.414387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.414359    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls" pod "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" (UID: "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:38:05.414478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.414379    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs" pod "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" (UID: "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:38:05.414478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.414439    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86" pod "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" (UID: "71319f1c-381b-4f8d-94c4-cc44a2e3a7d9"). InnerVolumeSpecName "kube-api-access-gtc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:05.513608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.513564    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:05.513608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.513602    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:05.513608 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.513615    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtc86\" (UniqueName: \"kubernetes.io/projected/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-kube-api-access-gtc86\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:05.513939 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.513629    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:05.725437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.720735    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 02 08:38:05.725437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.720803    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 02 08:38:05.746335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.741535    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:38:05.746335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:05.741609    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 02 08:38:06.207617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.207493    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"
Nov 02 08:38:06.207617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.207479    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx" event={"ID":"71319f1c-381b-4f8d-94c4-cc44a2e3a7d9","Type":"ContainerDied","Data":"76f6dcd160f9c408b711114ad2d0db0ed7ff7cda00b4e2c10a5684758e10d08f"}
Nov 02 08:38:06.207864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.207628    4515 scope.go:122] "RemoveContainer" containerID="28e32109839975369eee286d45058deae4fbeca6b4021fc306cd13461d60a2d5"
Nov 02 08:38:06.258507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.249580    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:38:06.258507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.258429    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:38:06.275805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.275769    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 08:38:06.288217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.288185    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-337764-6948dcb5bb-hm5gx"]
Nov 02 08:38:06.720888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.720845    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 02 08:38:06.721261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.721210    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 02 08:38:06.740877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.740843    4515 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:38:06.741168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.741126    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 02 08:38:06.990395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.987804    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:06.990395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.987998    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy" containerID="cri-o://5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704" gracePeriod=30
Nov 02 08:38:06.992374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.992349    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:06.992537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:06.992519    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy" containerID="cri-o://a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b" gracePeriod=30
Nov 02 08:38:07.376917 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:07.376878    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" path="/var/lib/kubelet/pods/71319f1c-381b-4f8d-94c4-cc44a2e3a7d9/volumes"
Nov 02 08:38:08.637811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:08.637775    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:38:13.771779 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.771751    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:13.812819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.812751    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:13.813128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813115    4515 cpu_manager.go:414] "containerMap: removing container" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy"
Nov 02 08:38:13.813173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813130    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy"
Nov 02 08:38:13.813173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813141    4515 cpu_manager.go:414] "containerMap: removing container" podUID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" containerName="maas-api"
Nov 02 08:38:13.813173 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813146    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" containerName="maas-api"
Nov 02 08:38:13.813275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813218    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" containerName="istio-proxy"
Nov 02 08:38:13.813275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.813227    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="71319f1c-381b-4f8d-94c4-cc44a2e3a7d9" containerName="maas-api"
Nov 02 08:38:13.817178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.817162    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826223    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826263    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826356    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826390    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826429    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826573    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826628    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826658    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.826731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.826698    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfsg\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.845337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.845007    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-337764-openshift-default-dockercfg-45fm2\""
Nov 02 08:38:13.847844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.847816    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:13.848446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.848426    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:13.927300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927271    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.927484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927324    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927361    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927394    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.927484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927423    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927458    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927488    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927524    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927553    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927585    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927590    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927635    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927668    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927691    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.927773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927720    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927865    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927910    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.927977    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928007    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928025    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928093    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928140    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928190    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.928271 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928220    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\") pod \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\" (UID: \"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc\") "
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928279    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\") pod \"56911157-12f1-4944-b4e9-130c59e7eec8\" (UID: \"56911157-12f1-4944-b4e9-130c59e7eec8\") "
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928372    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928384    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928446    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928489    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928519    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928560    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfsg\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928637    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928677    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928704    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928755    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928806    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928872    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928888    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56911157-12f1-4944-b4e9-130c59e7eec8-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928904    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928919    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928934    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928947    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928961    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928977    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.928993    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.929545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.929102    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.929728    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.930016    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.930458    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.930577    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.930645    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:38:13.930965 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.930748    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.931737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.931709    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.931871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.931761    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.931947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.931863    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:38:13.931947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.931896    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.932851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.932817    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.932851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.932845    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz" pod "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" (UID: "7a5ea1e3-e6c6-43fc-92c1-35c17088eadc"). InnerVolumeSpecName "kube-api-access-68xmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.933046 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.932995    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "kube-api-access-t9cml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.933172 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.933154    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.933522 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.933501    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy" pod "56911157-12f1-4944-b4e9-130c59e7eec8" (UID: "56911157-12f1-4944-b4e9-130c59e7eec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.952820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.952791    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:13.953176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:13.953155    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfsg\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:14.029884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029849    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.029884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029878    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.029884 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029887    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68xmz\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-kube-api-access-68xmz\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029899    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56911157-12f1-4944-b4e9-130c59e7eec8-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029908    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9cml\" (UniqueName: \"kubernetes.io/projected/56911157-12f1-4944-b4e9-130c59e7eec8-kube-api-access-t9cml\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029916    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029924    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56911157-12f1-4944-b4e9-130c59e7eec8-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029931    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.030124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.029940    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.132651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.132561    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:14.246035 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.246000    4515 generic.go:356] "Generic (PLEG): container finished" podID="56911157-12f1-4944-b4e9-130c59e7eec8" containerID="5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704" exitCode=0
Nov 02 08:38:14.246236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.246066    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"
Nov 02 08:38:14.246236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.246085    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" event={"ID":"56911157-12f1-4944-b4e9-130c59e7eec8","Type":"ContainerDied","Data":"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"}
Nov 02 08:38:14.246236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.246168    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8" event={"ID":"56911157-12f1-4944-b4e9-130c59e7eec8","Type":"ContainerDied","Data":"0dfdfe603dffd7293ebc0cb43b2aae19287971d7cf37df6bc634d1126f2dea8f"}
Nov 02 08:38:14.246236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.246192    4515 scope.go:122] "RemoveContainer" containerID="5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"
Nov 02 08:38:14.247614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.247583    4515 generic.go:356] "Generic (PLEG): container finished" podID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerID="a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b" exitCode=0
Nov 02 08:38:14.247754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.247616    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" event={"ID":"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc","Type":"ContainerDied","Data":"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"}
Nov 02 08:38:14.247754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.247641    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"
Nov 02 08:38:14.247754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.247667    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6" event={"ID":"7a5ea1e3-e6c6-43fc-92c1-35c17088eadc","Type":"ContainerDied","Data":"60f20b248bfff33c91f8db1cdf2800f932163ddd8d54ee6d8bf5c1b409832949"}
Nov 02 08:38:14.254950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.254930    4515 scope.go:122] "RemoveContainer" containerID="5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"
Nov 02 08:38:14.255292 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:14.255225    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704\": container with ID starting with 5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704 not found: ID does not exist" containerID="5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"
Nov 02 08:38:14.255292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.255249    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704"} err="failed to get container status \"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704\": rpc error: code = NotFound desc = could not find container \"5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704\": container with ID starting with 5ae21dff48d0fa4019ee5b498a5d13af1977c942a1426c1c87572a66d31b8704 not found: ID does not exist"
Nov 02 08:38:14.255292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.255266    4515 scope.go:122] "RemoveContainer" containerID="a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"
Nov 02 08:38:14.263425 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.263409    4515 scope.go:122] "RemoveContainer" containerID="a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"
Nov 02 08:38:14.263669 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:14.263651    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b\": container with ID starting with a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b not found: ID does not exist" containerID="a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"
Nov 02 08:38:14.263720 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.263678    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b"} err="failed to get container status \"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b\": rpc error: code = NotFound desc = could not find container \"a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b\": container with ID starting with a2aefdb17cb732240bfe8bbf3f8e8740344e1458ac57c34797d5a76eedd3c35b not found: ID does not exist"
Nov 02 08:38:14.298694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.298667    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:14.314049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.314015    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bzsf6"]
Nov 02 08:38:14.338090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.338055    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:14.341498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.341474    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-5plz8"]
Nov 02 08:38:14.446010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.445977    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:14.446303 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:14.446282    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3390a300_6ac0_48d8_89af_2e982716fe2d.slice/crio-c5c770ca1b6bdca1d7e906d325645b2df4ff50798feb63e9b19673900af904ff WatchSource:0}: Error finding container c5c770ca1b6bdca1d7e906d325645b2df4ff50798feb63e9b19673900af904ff: Status 404 returned error can't find the container with id c5c770ca1b6bdca1d7e906d325645b2df4ff50798feb63e9b19673900af904ff
Nov 02 08:38:14.448442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.448408    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:14.448531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.448475    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:14.448531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:14.448506    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:15.256773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:15.256732    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" event={"ID":"3390a300-6ac0-48d8-89af-2e982716fe2d","Type":"ContainerStarted","Data":"83adbf284ffd72ea3b41430609bac312d056d987e49a95499b014cfab5c9a62e"}
Nov 02 08:38:15.257272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:15.256801    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" event={"ID":"3390a300-6ac0-48d8-89af-2e982716fe2d","Type":"ContainerStarted","Data":"c5c770ca1b6bdca1d7e906d325645b2df4ff50798feb63e9b19673900af904ff"}
Nov 02 08:38:15.341381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:15.334125    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podStartSLOduration=2.334104964 podStartE2EDuration="2.334104964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:15.332842953 +0000 UTC m=+4440.095279859" watchObservedRunningTime="2026-11-02 08:38:15.334104964 +0000 UTC m=+4440.096541870"
Nov 02 08:38:15.371662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:15.371630    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56911157-12f1-4944-b4e9-130c59e7eec8" path="/var/lib/kubelet/pods/56911157-12f1-4944-b4e9-130c59e7eec8/volumes"
Nov 02 08:38:15.372084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:15.372070    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" path="/var/lib/kubelet/pods/7a5ea1e3-e6c6-43fc-92c1-35c17088eadc/volumes"
Nov 02 08:38:16.139294 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:16.139261    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:16.139717 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:16.139692    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 02 08:38:16.139892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:16.139868    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 02 08:38:16.242052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:16.242014    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:38:16.242626 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:16.242601    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:38:17.136846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.136801    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 02 08:38:17.137571 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.137538    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 02 08:38:17.687245 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.687216    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 08:38:17.715826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.715743    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 08:38:17.752475 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:17.752438    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:38:17.763319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763275    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\") pod \"af990a90-7f57-4df0-967f-2f662229390c\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") "
Nov 02 08:38:17.763520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763325    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\" (UniqueName: \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\") pod \"60ecc2e1-0598-4102-84fc-ae6b6303660d\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") "
Nov 02 08:38:17.763520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763352    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\") pod \"60ecc2e1-0598-4102-84fc-ae6b6303660d\" (UID: \"60ecc2e1-0598-4102-84fc-ae6b6303660d\") "
Nov 02 08:38:17.763520 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763404    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\" (UniqueName: \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\") pod \"af990a90-7f57-4df0-967f-2f662229390c\" (UID: \"af990a90-7f57-4df0-967f-2f662229390c\") "
Nov 02 08:38:17.763728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763697    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume" pod "af990a90-7f57-4df0-967f-2f662229390c" (UID: "af990a90-7f57-4df0-967f-2f662229390c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:17.763728 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.763718    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume" pod "60ecc2e1-0598-4102-84fc-ae6b6303660d" (UID: "60ecc2e1-0598-4102-84fc-ae6b6303660d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:17.765496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.765470    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h" pod "60ecc2e1-0598-4102-84fc-ae6b6303660d" (UID: "60ecc2e1-0598-4102-84fc-ae6b6303660d"). InnerVolumeSpecName "kube-api-access-vbt7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:17.765629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.765603    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp" pod "af990a90-7f57-4df0-967f-2f662229390c" (UID: "af990a90-7f57-4df0-967f-2f662229390c"). InnerVolumeSpecName "kube-api-access-b7bxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:17.864038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.863990    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af990a90-7f57-4df0-967f-2f662229390c-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:17.864038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.864035    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbt7h\" (UniqueName: \"kubernetes.io/projected/60ecc2e1-0598-4102-84fc-ae6b6303660d-kube-api-access-vbt7h\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:17.864038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.864045    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ecc2e1-0598-4102-84fc-ae6b6303660d-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:17.864338 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:17.864056    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7bxp\" (UniqueName: \"kubernetes.io/projected/af990a90-7f57-4df0-967f-2f662229390c-kube-api-access-b7bxp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:18.138334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.137776    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 02 08:38:18.138334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.137846    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 02 08:38:18.272924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.272887    4515 generic.go:356] "Generic (PLEG): container finished" podID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerID="059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28" exitCode=0
Nov 02 08:38:18.273131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.272931    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"
Nov 02 08:38:18.273131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.272970    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" event={"ID":"60ecc2e1-0598-4102-84fc-ae6b6303660d","Type":"ContainerDied","Data":"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"}
Nov 02 08:38:18.273131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.273027    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w" event={"ID":"60ecc2e1-0598-4102-84fc-ae6b6303660d","Type":"ContainerDied","Data":"3c71cf2ed32b32a18d7a24457a09622f28516279a78ead31cbebec9956aa4b23"}
Nov 02 08:38:18.273131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.273057    4515 scope.go:122] "RemoveContainer" containerID="059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"
Nov 02 08:38:18.274381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.274358    4515 generic.go:356] "Generic (PLEG): container finished" podID="af990a90-7f57-4df0-967f-2f662229390c" containerID="f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351" exitCode=0
Nov 02 08:38:18.274460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.274389    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"
Nov 02 08:38:18.274460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.274445    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" event={"ID":"af990a90-7f57-4df0-967f-2f662229390c","Type":"ContainerDied","Data":"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"}
Nov 02 08:38:18.274568 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.274499    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl" event={"ID":"af990a90-7f57-4df0-967f-2f662229390c","Type":"ContainerDied","Data":"c4b067a331adb3819043e0ad91ceaa3460949ab1dbbe5ca26c18ad83b32ab5b1"}
Nov 02 08:38:18.283562 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.283546    4515 scope.go:122] "RemoveContainer" containerID="059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"
Nov 02 08:38:18.283869 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:18.283844    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28\": container with ID starting with 059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28 not found: ID does not exist" containerID="059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"
Nov 02 08:38:18.283970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.283875    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28"} err="failed to get container status \"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28\": rpc error: code = NotFound desc = could not find container \"059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28\": container with ID starting with 059f8a22dd1ab140fad0ec4c1fef0990949a0b4e332fa686948bc4e6bb04be28 not found: ID does not exist"
Nov 02 08:38:18.283970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.283901    4515 scope.go:122] "RemoveContainer" containerID="f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"
Nov 02 08:38:18.292158 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.292139    4515 scope.go:122] "RemoveContainer" containerID="f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"
Nov 02 08:38:18.292464 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:18.292443    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351\": container with ID starting with f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351 not found: ID does not exist" containerID="f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"
Nov 02 08:38:18.292527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.292472    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351"} err="failed to get container status \"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351\": rpc error: code = NotFound desc = could not find container \"f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351\": container with ID starting with f986d9371abfbde169db77540541681d53a162caa95a9dfe51430d7f834e1351 not found: ID does not exist"
Nov 02 08:38:18.341411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.341139    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 08:38:18.353011 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.352966    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-337764-5fbd7f5d59-chd2w"]
Nov 02 08:38:18.381741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.380644    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 08:38:18.385699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:18.385651    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-337764-79b874659f-8bjsl"]
Nov 02 08:38:19.139332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:19.139096    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 02 08:38:19.139332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:19.139157    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 02 08:38:19.371619 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:19.371539    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" path="/var/lib/kubelet/pods/60ecc2e1-0598-4102-84fc-ae6b6303660d/volumes"
Nov 02 08:38:19.371928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:19.371915    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af990a90-7f57-4df0-967f-2f662229390c" path="/var/lib/kubelet/pods/af990a90-7f57-4df0-967f-2f662229390c/volumes"
Nov 02 08:38:20.138333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:20.137794    4515 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 02 08:38:20.138333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:20.137861    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 02 08:38:21.137157 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:21.137126    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:21.137653 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:21.137435    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:21.138067 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:21.138050    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:25.182544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:25.182508    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:25.185482 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:25.182733    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy" containerID="cri-o://83adbf284ffd72ea3b41430609bac312d056d987e49a95499b014cfab5c9a62e" gracePeriod=30
Nov 02 08:38:27.521094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:27.521061    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d5hvn_33c78a7e-3e83-4a53-ab31-75f0ef12ade5/limitador/0.log"
Nov 02 08:38:28.662626 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:28.662589    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-785d49bf89-46p4s_dc11f2c7-aac8-4213-aa09-42b98173d446/manager/0.log"
Nov 02 08:38:29.776992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:29.776953    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f6b5c786d-jb7pn_e21f8235-4923-4a54-98dd-164cac55fadd/manager/0.log"
Nov 02 08:38:30.328972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.328938    4515 generic.go:356] "Generic (PLEG): container finished" podID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerID="83adbf284ffd72ea3b41430609bac312d056d987e49a95499b014cfab5c9a62e" exitCode=0
Nov 02 08:38:30.329166 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.329018    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" event={"ID":"3390a300-6ac0-48d8-89af-2e982716fe2d","Type":"ContainerDied","Data":"83adbf284ffd72ea3b41430609bac312d056d987e49a95499b014cfab5c9a62e"}
Nov 02 08:38:30.351389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.351356    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5gs_cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df/proxy/0.log"
Nov 02 08:38:30.428304 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.428275    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:30.475483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475388    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475440    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475471    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475507    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475571    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475601    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475636    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475663    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.475787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475696    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\") pod \"3390a300-6ac0-48d8-89af-2e982716fe2d\" (UID: \"3390a300-6ac0-48d8-89af-2e982716fe2d\") "
Nov 02 08:38:30.476108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475842    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:30.476108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475929    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:30.476108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475947    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:30.476108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.475963    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:30.476108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476097    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:30.476412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476172    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.476412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476190    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3390a300-6ac0-48d8-89af-2e982716fe2d-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.476412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476204    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.476412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476219    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.476412 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.476234    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.478033 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.478001    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "kube-api-access-rnfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:30.478168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.478086    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:38:30.478168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.478161    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:30.478290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.478255    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token" pod "3390a300-6ac0-48d8-89af-2e982716fe2d" (UID: "3390a300-6ac0-48d8-89af-2e982716fe2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:30.577462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.577420    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.577462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.577455    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.577462 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.577468    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfsg\" (UniqueName: \"kubernetes.io/projected/3390a300-6ac0-48d8-89af-2e982716fe2d-kube-api-access-rnfsg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:30.577806 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:30.577482    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3390a300-6ac0-48d8-89af-2e982716fe2d-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:38:31.335484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:31.335448    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"
Nov 02 08:38:31.335973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:31.335447    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn" event={"ID":"3390a300-6ac0-48d8-89af-2e982716fe2d","Type":"ContainerDied","Data":"c5c770ca1b6bdca1d7e906d325645b2df4ff50798feb63e9b19673900af904ff"}
Nov 02 08:38:31.335973 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:31.335594    4515 scope.go:122] "RemoveContainer" containerID="83adbf284ffd72ea3b41430609bac312d056d987e49a95499b014cfab5c9a62e"
Nov 02 08:38:31.402753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:31.402710    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:31.415201 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:31.415169    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-ltkqn"]
Nov 02 08:38:33.370962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:33.370928    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" path="/var/lib/kubelet/pods/3390a300-6ac0-48d8-89af-2e982716fe2d/volumes"
Nov 02 08:38:35.402336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.402164    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.402895    4515 cpu_manager.go:414] "containerMap: removing container" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.402938    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.402971    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.402980    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403019    4515 cpu_manager.go:414] "containerMap: removing container" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403026    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403041    4515 cpu_manager.go:414] "containerMap: removing container" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403049    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403224    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3390a300-6ac0-48d8-89af-2e982716fe2d" containerName="istio-proxy"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403238    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ecc2e1-0598-4102-84fc-ae6b6303660d" containerName="payload-pre-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403258    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="af990a90-7f57-4df0-967f-2f662229390c" containerName="payload-processing"
Nov 02 08:38:35.406105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.403275    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5ea1e3-e6c6-43fc-92c1-35c17088eadc" containerName="istio-proxy"
Nov 02 08:38:35.408498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.408474    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.426103 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.426077    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-466229b7-openshift-default-dockercfg-md8vd\""
Nov 02 08:38:35.438713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.438681    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 08:38:35.440525 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.440497    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:38:35.524442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524403    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524457    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524601    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524637    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524674    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrp2\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524789    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524839    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524924    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.524992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.524955    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.625874 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.625832    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.625877    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.625919    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrp2\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.625956    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.625994    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626035    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626060    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626091    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626342    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626469 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626349    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626531    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626596    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626707    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.626796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.626771    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.628322 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.628292    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.628565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.628547    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.649507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.649479    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.649651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.649559    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrp2\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\") pod \"e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") " pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:35.721218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:35.721139    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:36.035529 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:36.035500    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f71e9ba_ff73_4f59_a54a_7dcbf65786ab.slice/crio-df9e15c9bdd7b2fc5e50127f2c46357e623c25b4fee1ad682a2a6f7eba3a2da4 WatchSource:0}: Error finding container df9e15c9bdd7b2fc5e50127f2c46357e623c25b4fee1ad682a2a6f7eba3a2da4: Status 404 returned error can't find the container with id df9e15c9bdd7b2fc5e50127f2c46357e623c25b4fee1ad682a2a6f7eba3a2da4
Nov 02 08:38:36.035743 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.035721    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:38:36.037654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.037618    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:36.037755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.037686    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:36.037755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.037714    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:36.361869 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.361776    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerStarted","Data":"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"}
Nov 02 08:38:36.361869 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.361838    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerStarted","Data":"df9e15c9bdd7b2fc5e50127f2c46357e623c25b4fee1ad682a2a6f7eba3a2da4"}
Nov 02 08:38:36.409946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.409889    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podStartSLOduration=1.409876712 podStartE2EDuration="1.409876712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:36.408232093 +0000 UTC m=+4461.170668998" watchObservedRunningTime="2026-11-02 08:38:36.409876712 +0000 UTC m=+4461.172313609"
Nov 02 08:38:36.721963 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:36.721885    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:37.727115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:37.727076    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:38.371325 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:38.371283    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:38.372369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:38.372329    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:39.650350 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.650300    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:38:39.653646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.653629    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.682731 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.682695    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-466229b7-serving-cert\""
Nov 02 08:38:39.685326 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.685285    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:38:39.766793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.766762    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.766986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.766817    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kd54\" (UniqueName: \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.766986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.766967    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.767063 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.767022    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.867823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.867787    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.868024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.867846    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kd54\" (UniqueName: \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.868024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.867904    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.868024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.867939    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.868577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.868555    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.870464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.870443    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.870582 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.870507    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.890867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.890841    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kd54\" (UniqueName: \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\") pod \"maas-api-e2e-ait-466229b7-5845f9d9db-9558d\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:39.936640 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.936564    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:38:39.940198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.940180    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:39.959298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.959271    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-466229b7\""
Nov 02 08:38:39.959468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.959347    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-466229b7-dockercfg-kqk4d\""
Nov 02 08:38:39.961002 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.960980    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:38:39.966455 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:39.966436    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:40.035225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.033425    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:38:40.041346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.041185    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.058092 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.058056    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:38:40.069683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.069647    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.069839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.069746    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fjl\" (UniqueName: \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.170457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.170423    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.170668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.170479    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fjl\" (UniqueName: \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.170668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.170544    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.170668 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.170570    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8vg\" (UniqueName: \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.171191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.171174    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.193943 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.193872    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fjl\" (UniqueName: \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\") pod \"payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.253280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.253247    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:40.265678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.265652    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:38:40.267064 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:40.267034    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c720a2_d0af_449b_9a7c_940174d159da.slice/crio-3dca2660fc8c75eb1444d8abaec6f5dedb6428e9dd6d0a369d2e00ceee20b994 WatchSource:0}: Error finding container 3dca2660fc8c75eb1444d8abaec6f5dedb6428e9dd6d0a369d2e00ceee20b994: Status 404 returned error can't find the container with id 3dca2660fc8c75eb1444d8abaec6f5dedb6428e9dd6d0a369d2e00ceee20b994
Nov 02 08:38:40.268545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.268527    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 08:38:40.271808 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.271779    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.271912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.271811    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8vg\" (UniqueName: \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.272505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.272482    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.295872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.295838    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8vg\" (UniqueName: \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\") pod \"payload-processing-e2e-ait-466229b7-757d899967-mqk4x\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") " pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.358233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.358195    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:40.379496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.379463    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" event={"ID":"b1c720a2-d0af-449b-9a7c-940174d159da","Type":"ContainerStarted","Data":"3dca2660fc8c75eb1444d8abaec6f5dedb6428e9dd6d0a369d2e00ceee20b994"}
Nov 02 08:38:40.553400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.553360    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:38:40.553745 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:40.553722    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281aed5c_9fd6_43d1_b199_83c15b38e5b1.slice/crio-c58a85fd73ed6548f802232597e09cb6959b0d180d4e292331a7b9209e738e32 WatchSource:0}: Error finding container c58a85fd73ed6548f802232597e09cb6959b0d180d4e292331a7b9209e738e32: Status 404 returned error can't find the container with id c58a85fd73ed6548f802232597e09cb6959b0d180d4e292331a7b9209e738e32
Nov 02 08:38:40.659062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:40.659029    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:38:40.660007 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:38:40.659980    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7531b8_d52c_4e63_a871_c895e87b5d8e.slice/crio-12ef74664c2912b0eebdffe37b3363c38ec910e0a8c6f27db517c0245dbd31ae WatchSource:0}: Error finding container 12ef74664c2912b0eebdffe37b3363c38ec910e0a8c6f27db517c0245dbd31ae: Status 404 returned error can't find the container with id 12ef74664c2912b0eebdffe37b3363c38ec910e0a8c6f27db517c0245dbd31ae
Nov 02 08:38:41.384770 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.384731    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" event={"ID":"b1c720a2-d0af-449b-9a7c-940174d159da","Type":"ContainerStarted","Data":"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"}
Nov 02 08:38:41.384947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.384902    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:41.386287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.386261    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" event={"ID":"ec7531b8-d52c-4e63-a871-c895e87b5d8e","Type":"ContainerStarted","Data":"e9604020d0572ac6282138678932d73f469a8758d2fdd55230c7d869a171ade7"}
Nov 02 08:38:41.386437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.386331    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" event={"ID":"ec7531b8-d52c-4e63-a871-c895e87b5d8e","Type":"ContainerStarted","Data":"12ef74664c2912b0eebdffe37b3363c38ec910e0a8c6f27db517c0245dbd31ae"}
Nov 02 08:38:41.386437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.386362    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:41.387842 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.387820    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" event={"ID":"281aed5c-9fd6-43d1-b199-83c15b38e5b1","Type":"ContainerStarted","Data":"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"}
Nov 02 08:38:41.387959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.387855    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" event={"ID":"281aed5c-9fd6-43d1-b199-83c15b38e5b1","Type":"ContainerStarted","Data":"c58a85fd73ed6548f802232597e09cb6959b0d180d4e292331a7b9209e738e32"}
Nov 02 08:38:41.387959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.387958    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:41.432226 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.432177    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" podStartSLOduration=1.9210362650000001 podStartE2EDuration="2.432161933s" totalImagesPullingTime="511.125668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:41.426932227 +0000 UTC m=+4466.189369132" watchObservedRunningTime="2026-11-02 08:38:41.432161933 +0000 UTC m=+4466.194598838"
Nov 02 08:38:41.502738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.502678    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" podStartSLOduration=1.502665612 podStartE2EDuration="1.502665612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:41.501120388 +0000 UTC m=+4466.263557293" watchObservedRunningTime="2026-11-02 08:38:41.502665612 +0000 UTC m=+4466.265102518"
Nov 02 08:38:41.560844 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:41.560788    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" podStartSLOduration=2.560774387 podStartE2EDuration="2.560774387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:41.560210545 +0000 UTC m=+4466.322647455" watchObservedRunningTime="2026-11-02 08:38:41.560774387 +0000 UTC m=+4466.323211292"
Nov 02 08:38:43.532735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:43.532697    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:38:47.399475 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:47.399445    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:38:47.744401 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:38:47.744285    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:38:52.399051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:52.397699    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:38:52.399051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:52.397914    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:38:53.412609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.412439    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": read tcp 10.133.0.2:35852->10.133.1.60:15021: read: connection reset by peer" start-of-body=
Nov 02 08:38:53.412609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.412508    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": read tcp 10.133.0.2:35852->10.133.1.60:15021: read: connection reset by peer"
Nov 02 08:38:53.471456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.471422    4515 generic.go:356] "Generic (PLEG): container finished" podID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerID="5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82" exitCode=0
Nov 02 08:38:53.471641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.471488    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerDied","Data":"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"}
Nov 02 08:38:53.471858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.471845    4515 scope.go:122] "RemoveContainer" containerID="5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"
Nov 02 08:38:53.473024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.472993    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:53.473095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.473064    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:53.473148 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:53.473095    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:54.478451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:54.478418    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerStarted","Data":"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"}
Nov 02 08:38:54.736388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:54.735772    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:54.736388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:54.735930    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 08:38:54.736388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:54.735980    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 08:38:55.724335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:55.721693    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 08:38:55.724335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:55.721756    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 08:38:56.731729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:56.731688    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 08:38:56.732378 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:56.732347    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 08:38:57.733732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:57.733693    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 08:38:57.734386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:57.734346    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 08:38:58.738335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:58.729771    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 08:38:58.738335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:58.729839    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 08:38:59.725746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:59.725714    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:59.726131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:59.726104    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:38:59.726890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:38:59.726867    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:39:04.426762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:04.426726    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:39:04.463324 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:04.463283    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:39:05.150098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.150057    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:39:05.150413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.150379    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" podUID="b1c720a2-d0af-449b-9a7c-940174d159da" containerName="maas-api" containerID="cri-o://eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932" gracePeriod=30
Nov 02 08:39:05.404186 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.404122    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:39:05.444656 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.444626    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\" (UniqueName: \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\") pod \"b1c720a2-d0af-449b-9a7c-940174d159da\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") "
Nov 02 08:39:05.445179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.444736    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\") pod \"b1c720a2-d0af-449b-9a7c-940174d159da\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") "
Nov 02 08:39:05.445179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.444774    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\") pod \"b1c720a2-d0af-449b-9a7c-940174d159da\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") "
Nov 02 08:39:05.445179 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.444812    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\") pod \"b1c720a2-d0af-449b-9a7c-940174d159da\" (UID: \"b1c720a2-d0af-449b-9a7c-940174d159da\") "
Nov 02 08:39:05.445433 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.445201    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca" pod "b1c720a2-d0af-449b-9a7c-940174d159da" (UID: "b1c720a2-d0af-449b-9a7c-940174d159da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:39:05.447288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.447245    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs" pod "b1c720a2-d0af-449b-9a7c-940174d159da" (UID: "b1c720a2-d0af-449b-9a7c-940174d159da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:05.447470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.447357    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54" pod "b1c720a2-d0af-449b-9a7c-940174d159da" (UID: "b1c720a2-d0af-449b-9a7c-940174d159da"). InnerVolumeSpecName "kube-api-access-2kd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:05.449513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.449488    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls" pod "b1c720a2-d0af-449b-9a7c-940174d159da" (UID: "b1c720a2-d0af-449b-9a7c-940174d159da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:05.529813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.529776    4515 generic.go:356] "Generic (PLEG): container finished" podID="b1c720a2-d0af-449b-9a7c-940174d159da" containerID="eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932" exitCode=0
Nov 02 08:39:05.530021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.529839    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"
Nov 02 08:39:05.530021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.529859    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" event={"ID":"b1c720a2-d0af-449b-9a7c-940174d159da","Type":"ContainerDied","Data":"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"}
Nov 02 08:39:05.530021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.529922    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d" event={"ID":"b1c720a2-d0af-449b-9a7c-940174d159da","Type":"ContainerDied","Data":"3dca2660fc8c75eb1444d8abaec6f5dedb6428e9dd6d0a369d2e00ceee20b994"}
Nov 02 08:39:05.530021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.529952    4515 scope.go:122] "RemoveContainer" containerID="eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"
Nov 02 08:39:05.539581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.539549    4515 scope.go:122] "RemoveContainer" containerID="eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"
Nov 02 08:39:05.539886 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:05.539863    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932\": container with ID starting with eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932 not found: ID does not exist" containerID="eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"
Nov 02 08:39:05.539947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.539895    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932"} err="failed to get container status \"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932\": rpc error: code = NotFound desc = could not find container \"eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932\": container with ID starting with eb06c2489a8a930c5dff1de2ef18e3d8285e51c8f87b24cc8631be33ebfd6932 not found: ID does not exist"
Nov 02 08:39:05.545923 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.545898    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kd54\" (UniqueName: \"kubernetes.io/projected/b1c720a2-d0af-449b-9a7c-940174d159da-kube-api-access-2kd54\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:05.545923 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.545924    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c720a2-d0af-449b-9a7c-940174d159da-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:05.546079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.545934    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:05.546079 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.545944    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c720a2-d0af-449b-9a7c-940174d159da-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:05.581652 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.581624    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:39:05.588914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:05.588886    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-466229b7-5845f9d9db-9558d"]
Nov 02 08:39:07.372027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:07.371987    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c720a2-d0af-449b-9a7c-940174d159da" path="/var/lib/kubelet/pods/b1c720a2-d0af-449b-9a7c-940174d159da/volumes"
Nov 02 08:39:07.422810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:07.422773    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:39:09.427838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.427791    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" podUID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" containerName="payload-processing" containerID="cri-o://e9604020d0572ac6282138678932d73f469a8758d2fdd55230c7d869a171ade7" gracePeriod=25
Nov 02 08:39:09.464474 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.464421    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" podUID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" containerName="payload-pre-processing" containerID="cri-o://ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc" gracePeriod=25
Nov 02 08:39:09.549354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.549319    4515 generic.go:356] "Generic (PLEG): container finished" podID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" containerID="e9604020d0572ac6282138678932d73f469a8758d2fdd55230c7d869a171ade7" exitCode=0
Nov 02 08:39:09.549354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.549338    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" event={"ID":"ec7531b8-d52c-4e63-a871-c895e87b5d8e","Type":"ContainerDied","Data":"e9604020d0572ac6282138678932d73f469a8758d2fdd55230c7d869a171ade7"}
Nov 02 08:39:09.704572 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.704546    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:39:09.733853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.733831    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:39:09.781358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781327    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\" (UniqueName: \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\") pod \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") "
Nov 02 08:39:09.781538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781500    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\") pod \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") "
Nov 02 08:39:09.781598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781567    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\") pod \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\" (UID: \"281aed5c-9fd6-43d1-b199-83c15b38e5b1\") "
Nov 02 08:39:09.781598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781591    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\" (UniqueName: \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\") pod \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\" (UID: \"ec7531b8-d52c-4e63-a871-c895e87b5d8e\") "
Nov 02 08:39:09.781946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781917    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume" pod "ec7531b8-d52c-4e63-a871-c895e87b5d8e" (UID: "ec7531b8-d52c-4e63-a871-c895e87b5d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:39:09.782082 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.781924    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume" pod "281aed5c-9fd6-43d1-b199-83c15b38e5b1" (UID: "281aed5c-9fd6-43d1-b199-83c15b38e5b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:39:09.783529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.783504    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl" pod "281aed5c-9fd6-43d1-b199-83c15b38e5b1" (UID: "281aed5c-9fd6-43d1-b199-83c15b38e5b1"). InnerVolumeSpecName "kube-api-access-r4fjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:09.783689 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.783670    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg" pod "ec7531b8-d52c-4e63-a871-c895e87b5d8e" (UID: "ec7531b8-d52c-4e63-a871-c895e87b5d8e"). InnerVolumeSpecName "kube-api-access-bs8vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:09.882596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.882556    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281aed5c-9fd6-43d1-b199-83c15b38e5b1-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:09.882596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.882589    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8vg\" (UniqueName: \"kubernetes.io/projected/ec7531b8-d52c-4e63-a871-c895e87b5d8e-kube-api-access-bs8vg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:09.882596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.882598    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4fjl\" (UniqueName: \"kubernetes.io/projected/281aed5c-9fd6-43d1-b199-83c15b38e5b1-kube-api-access-r4fjl\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:09.882596 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:09.882607    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7531b8-d52c-4e63-a871-c895e87b5d8e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:10.150994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.150958    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:39:10.151238 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.151213    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" containerID="cri-o://a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff" gracePeriod=30
Nov 02 08:39:10.554764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.554732    4515 generic.go:356] "Generic (PLEG): container finished" podID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" containerID="ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc" exitCode=0
Nov 02 08:39:10.555221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.554777    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"
Nov 02 08:39:10.555221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.554824    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" event={"ID":"281aed5c-9fd6-43d1-b199-83c15b38e5b1","Type":"ContainerDied","Data":"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"}
Nov 02 08:39:10.555221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.554882    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj" event={"ID":"281aed5c-9fd6-43d1-b199-83c15b38e5b1","Type":"ContainerDied","Data":"c58a85fd73ed6548f802232597e09cb6959b0d180d4e292331a7b9209e738e32"}
Nov 02 08:39:10.555221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.554906    4515 scope.go:122] "RemoveContainer" containerID="ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"
Nov 02 08:39:10.556453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.556428    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x" event={"ID":"ec7531b8-d52c-4e63-a871-c895e87b5d8e","Type":"ContainerDied","Data":"12ef74664c2912b0eebdffe37b3363c38ec910e0a8c6f27db517c0245dbd31ae"}
Nov 02 08:39:10.556569 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.556438    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"
Nov 02 08:39:10.566886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.566718    4515 scope.go:122] "RemoveContainer" containerID="ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"
Nov 02 08:39:10.567273 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:10.567253    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc\": container with ID starting with ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc not found: ID does not exist" containerID="ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"
Nov 02 08:39:10.567366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.567280    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc"} err="failed to get container status \"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc\": rpc error: code = NotFound desc = could not find container \"ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc\": container with ID starting with ab30e6a538e5d21aaddab16beae44c95cb5db376cbcfd43b5ae0e3fdca1e6fdc not found: ID does not exist"
Nov 02 08:39:10.567366 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.567298    4515 scope.go:122] "RemoveContainer" containerID="e9604020d0572ac6282138678932d73f469a8758d2fdd55230c7d869a171ade7"
Nov 02 08:39:10.609883 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.609855    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:39:10.614449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.614422    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-466229b7-757d899967-mqk4x"]
Nov 02 08:39:10.648866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.648812    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:39:10.652395 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:10.652372    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-466229b7-747d765b4c-p2flj"]
Nov 02 08:39:11.371230 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:11.371193    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" path="/var/lib/kubelet/pods/281aed5c-9fd6-43d1-b199-83c15b38e5b1/volumes"
Nov 02 08:39:11.371657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:11.371643    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" path="/var/lib/kubelet/pods/ec7531b8-d52c-4e63-a871-c895e87b5d8e/volumes"
Nov 02 08:39:14.039986 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.039951    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 08:39:14.040471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.040158    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" podUID="e21f8235-4923-4a54-98dd-164cac55fadd" containerName="manager" containerID="cri-o://3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45" gracePeriod=30
Nov 02 08:39:14.283566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.283540    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 08:39:14.424464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.424373    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\") pod \"e21f8235-4923-4a54-98dd-164cac55fadd\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") "
Nov 02 08:39:14.424464 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.424462    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\") pod \"e21f8235-4923-4a54-98dd-164cac55fadd\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") "
Nov 02 08:39:14.424680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.424501    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\" (UniqueName: \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\") pod \"e21f8235-4923-4a54-98dd-164cac55fadd\" (UID: \"e21f8235-4923-4a54-98dd-164cac55fadd\") "
Nov 02 08:39:14.426679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.426644    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert" pod "e21f8235-4923-4a54-98dd-164cac55fadd" (UID: "e21f8235-4923-4a54-98dd-164cac55fadd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:14.426679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.426670    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs" pod "e21f8235-4923-4a54-98dd-164cac55fadd" (UID: "e21f8235-4923-4a54-98dd-164cac55fadd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:14.426846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.426642    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r" pod "e21f8235-4923-4a54-98dd-164cac55fadd" (UID: "e21f8235-4923-4a54-98dd-164cac55fadd"). InnerVolumeSpecName "kube-api-access-6kk5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:14.525198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.525157    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kk5r\" (UniqueName: \"kubernetes.io/projected/e21f8235-4923-4a54-98dd-164cac55fadd-kube-api-access-6kk5r\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:14.525198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.525187    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:14.525198 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.525198    4515 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e21f8235-4923-4a54-98dd-164cac55fadd-webhook-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:14.584618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.584578    4515 generic.go:356] "Generic (PLEG): container finished" podID="e21f8235-4923-4a54-98dd-164cac55fadd" containerID="3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45" exitCode=0
Nov 02 08:39:14.584845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.584628    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn"
Nov 02 08:39:14.584845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.584661    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" event={"ID":"e21f8235-4923-4a54-98dd-164cac55fadd","Type":"ContainerDied","Data":"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"}
Nov 02 08:39:14.584845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.584729    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-jb7pn" event={"ID":"e21f8235-4923-4a54-98dd-164cac55fadd","Type":"ContainerDied","Data":"80ba6ccb988c8076999bc5e1bbf292bf843004bb56210d521ddee823121c41e7"}
Nov 02 08:39:14.584845 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.584760    4515 scope.go:122] "RemoveContainer" containerID="3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"
Nov 02 08:39:14.595885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.595866    4515 scope.go:122] "RemoveContainer" containerID="3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"
Nov 02 08:39:14.596175 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:14.596156    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45\": container with ID starting with 3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45 not found: ID does not exist" containerID="3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"
Nov 02 08:39:14.596257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.596181    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45"} err="failed to get container status \"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45\": rpc error: code = NotFound desc = could not find container \"3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45\": container with ID starting with 3e9ebb191a1d5b7804a09bfa6e7b36d14ffdb2731647de2cc394df285db9bc45 not found: ID does not exist"
Nov 02 08:39:14.638193 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.638160    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 08:39:14.650502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.650473    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-jb7pn"]
Nov 02 08:39:14.728509 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.728420    4515 patch_prober.go:29] interesting pod/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:39:14.728509 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:14.728493    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:39:15.341060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341027    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341454    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e21f8235-4923-4a54-98dd-164cac55fadd" containerName="manager"
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341467    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21f8235-4923-4a54-98dd-164cac55fadd" containerName="manager"
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341484    4515 cpu_manager.go:414] "containerMap: removing container" podUID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" containerName="payload-pre-processing"
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341490    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" containerName="payload-pre-processing"
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341506    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" containerName="payload-processing"
Nov 02 08:39:15.341506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341511    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" containerName="payload-processing"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341525    4515 cpu_manager.go:414] "containerMap: removing container" podUID="b1c720a2-d0af-449b-9a7c-940174d159da" containerName="maas-api"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341532    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c720a2-d0af-449b-9a7c-940174d159da" containerName="maas-api"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341628    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c720a2-d0af-449b-9a7c-940174d159da" containerName="maas-api"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341645    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21f8235-4923-4a54-98dd-164cac55fadd" containerName="manager"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341656    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="281aed5c-9fd6-43d1-b199-83c15b38e5b1" containerName="payload-pre-processing"
Nov 02 08:39:15.341762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.341667    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7531b8-d52c-4e63-a871-c895e87b5d8e" containerName="payload-processing"
Nov 02 08:39:15.345057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.345035    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.362602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.362578    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 08:39:15.373000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.372973    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21f8235-4923-4a54-98dd-164cac55fadd" path="/var/lib/kubelet/pods/e21f8235-4923-4a54-98dd-164cac55fadd/volumes"
Nov 02 08:39:15.376615 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.376592    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:15.408122 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.408084    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:39:15.408403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.408386    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 08:39:15.408403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.408388    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 08:39:15.408541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.408458    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xfsgv\""
Nov 02 08:39:15.432554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.432473    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvcx\" (UniqueName: \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.432554 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.432544    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.432744 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.432721    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.533618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533581    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.533833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533629    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.533833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533675    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.533976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533858    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.533976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533917    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.533976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533869    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:39:15.534145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.533993    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.534145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534029    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.534145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534074    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.534145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534087    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:39:15.534145 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534127    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\") pod \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\" (UID: \"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab\") "
Nov 02 08:39:15.534470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534199    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:39:15.534470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534342    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvcx\" (UniqueName: \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.534470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534394    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:39:15.534470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534427    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.534694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534475    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:39:15.534694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534545    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.534694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534647    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.534694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534666    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.534694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534682    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.534964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534695    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.534964 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.534708    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.536099 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.536074    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:39:15.536468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.536449    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:15.536561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.536507    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "kube-api-access-9mrp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:15.536561 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.536545    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo" pod "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" (UID: "8f71e9ba-ff73-4f59-a54a-7dcbf65786ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:39:15.548645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.548621    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 08:39:15.548764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.548704    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 08:39:15.557120 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.557085    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.557220 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.557097    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.559626 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.559605    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvcx\" (UniqueName: \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\") pod \"maas-controller-5f6b5c786d-hmrvb\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") " pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.592548    4515 generic.go:356] "Generic (PLEG): container finished" podID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerID="a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff" exitCode=0
Nov 02 08:39:15.592814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.592607    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"
Nov 02 08:39:15.592814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.592642    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerDied","Data":"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"}
Nov 02 08:39:15.592814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.592701    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f" event={"ID":"8f71e9ba-ff73-4f59-a54a-7dcbf65786ab","Type":"ContainerDied","Data":"df9e15c9bdd7b2fc5e50127f2c46357e623c25b4fee1ad682a2a6f7eba3a2da4"}
Nov 02 08:39:15.592814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.592741    4515 scope.go:122] "RemoveContainer" containerID="a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"
Nov 02 08:39:15.603066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.603039    4515 scope.go:122] "RemoveContainer" containerID="5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"
Nov 02 08:39:15.611144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.611124    4515 scope.go:122] "RemoveContainer" containerID="a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"
Nov 02 08:39:15.611444 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:15.611424    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff\": container with ID starting with a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff not found: ID does not exist" containerID="a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"
Nov 02 08:39:15.611512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.611452    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff"} err="failed to get container status \"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff\": rpc error: code = NotFound desc = could not find container \"a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff\": container with ID starting with a34f18102910796f18769e3fecc9c3b3ace7c3d8642567acd70fc7d979c461ff not found: ID does not exist"
Nov 02 08:39:15.611512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.611470    4515 scope.go:122] "RemoveContainer" containerID="5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"
Nov 02 08:39:15.611703 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:15.611684    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82\": container with ID starting with 5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82 not found: ID does not exist" containerID="5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"
Nov 02 08:39:15.611758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.611713    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82"} err="failed to get container status \"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82\": rpc error: code = NotFound desc = could not find container \"5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82\": container with ID starting with 5f4ad100edca41c397584631e7e7fac9596de5d1cd5420586507ad66a43dcd82 not found: ID does not exist"
Nov 02 08:39:15.641971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.636334    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.641971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.636402    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.641971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.636419    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.641971 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.636447    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrp2\" (UniqueName: \"kubernetes.io/projected/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab-kube-api-access-9mrp2\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:15.645409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.645378    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:39:15.649726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.649698    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-466229b7-openshift-default-85b7688cd5-4k67f"]
Nov 02 08:39:15.673591 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.673557    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xfsgv\""
Nov 02 08:39:15.679708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.679683    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:15.979660 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:15.979632    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:15.980173 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:39:15.980151    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798484b6_1710_4d0f_80d9_ebf7a690c259.slice/crio-eb1e3b0d6f74c797ada2e516aa3203b8821b2993edd46a62192cbf3b0550357b WatchSource:0}: Error finding container eb1e3b0d6f74c797ada2e516aa3203b8821b2993edd46a62192cbf3b0550357b: Status 404 returned error can't find the container with id eb1e3b0d6f74c797ada2e516aa3203b8821b2993edd46a62192cbf3b0550357b
Nov 02 08:39:16.462501 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:16.462471    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 08:39:16.599456 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:16.599420    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" event={"ID":"798484b6-1710-4d0f-80d9-ebf7a690c259","Type":"ContainerStarted","Data":"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"}
Nov 02 08:39:16.599662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:16.599473    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" event={"ID":"798484b6-1710-4d0f-80d9-ebf7a690c259","Type":"ContainerStarted","Data":"eb1e3b0d6f74c797ada2e516aa3203b8821b2993edd46a62192cbf3b0550357b"}
Nov 02 08:39:16.599662 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:16.599524    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:16.643732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:16.643685    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" podStartSLOduration=1.177338746 podStartE2EDuration="1.643673452s" totalImagesPullingTime="466.334706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:16.643040907 +0000 UTC m=+4501.405477812" watchObservedRunningTime="2026-11-02 08:39:16.643673452 +0000 UTC m=+4501.406110358"
Nov 02 08:39:17.371385 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:17.371346    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" path="/var/lib/kubelet/pods/8f71e9ba-ff73-4f59-a54a-7dcbf65786ab/volumes"
Nov 02 08:39:17.722752 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:17.722672    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:39:27.607686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:27.607644    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:28.448905 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.448872    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"]
Nov 02 08:39:28.449246 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.449231    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.449299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.449246    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.449377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.449330    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.449377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.449341    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.459439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.459408    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.465117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.465079    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"]
Nov 02 08:39:28.465529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.465506    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.465529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.465526    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f71e9ba-ff73-4f59-a54a-7dcbf65786ab" containerName="istio-proxy"
Nov 02 08:39:28.468373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.468354    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"]
Nov 02 08:39:28.468490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.468470    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.471357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.471342    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.477112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.477085    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 08:39:28.477249 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.477227    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7cxdc\""
Nov 02 08:39:28.478782 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.478751    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"]
Nov 02 08:39:28.488187 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.488162    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"]
Nov 02 08:39:28.491651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.491629    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.495957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.495930    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"]
Nov 02 08:39:28.498185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.498161    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"]
Nov 02 08:39:28.519121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.519088    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"]
Nov 02 08:39:28.553434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553403    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650daaeb-7c93-4844-8659-6bebaa064303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.553434 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553437    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553457    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.553657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553514    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553599    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4277s\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-kube-api-access-4277s\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553620    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbd9\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-kube-api-access-7xbd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.553657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553645    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553663    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553779    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553813    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553860    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553889    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72r55\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-kube-api-access-72r55\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.553948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553924    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.553979    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554016    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554051    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554071    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554111    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3629c3ef-7d97-4239-a095-71741d6885af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554162    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554193    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554247 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554220    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650daaeb-7c93-4844-8659-6bebaa064303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554254    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554325    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554350    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3629c3ef-7d97-4239-a095-71741d6885af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554365    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554385    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.554638 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.554455    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.655936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.655900    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.655943    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656017    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656049    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656069    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656098    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm99g\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-kube-api-access-fm99g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656118    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656135    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72r55\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-kube-api-access-72r55\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656184    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656207    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656231    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656260    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656301    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656363    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656388    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3629c3ef-7d97-4239-a095-71741d6885af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656424    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.656470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656451    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656476    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650daaeb-7c93-4844-8659-6bebaa064303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656545    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656578    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656608    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656620    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656675    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656725    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3629c3ef-7d97-4239-a095-71741d6885af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656753    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656780    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656812    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656836    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656856    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650daaeb-7c93-4844-8659-6bebaa064303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656869    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656881    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656919    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656957    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.656989    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657037    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4277s\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-kube-api-access-4277s\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657070    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbd9\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-kube-api-access-7xbd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657098    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657123    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657134    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657161    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657197    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f9d878-fb9d-437c-b81d-defc288cc2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657303    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650daaeb-7c93-4844-8659-6bebaa064303-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.657418 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657423    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657585    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657679    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657771    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.658461    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.658723    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3629c3ef-7d97-4239-a095-71741d6885af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.658743    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.658882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.657161    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.659257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.658982    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.659257 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.659190    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.659447 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.659431    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.659681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.659663    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650daaeb-7c93-4844-8659-6bebaa064303-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.659852 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.659835    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3629c3ef-7d97-4239-a095-71741d6885af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.660058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.660035    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3629c3ef-7d97-4239-a095-71741d6885af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.660893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.660874    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650daaeb-7c93-4844-8659-6bebaa064303-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.679787 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.679761    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.680067 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.680043    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72r55\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-kube-api-access-72r55\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.680178 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.680100    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.681153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.681125    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3629c3ef-7d97-4239-a095-71741d6885af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jl6c9\" (UID: \"3629c3ef-7d97-4239-a095-71741d6885af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.681254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.681195    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbd9\" (UniqueName: \"kubernetes.io/projected/650daaeb-7c93-4844-8659-6bebaa064303-kube-api-access-7xbd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-66v28\" (UID: \"650daaeb-7c93-4844-8659-6bebaa064303\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.681467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.681449    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4277s\" (UniqueName: \"kubernetes.io/projected/a63a70fa-8feb-47e4-8cef-fd3b1b234b5a-kube-api-access-4277s\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wgbb7\" (UID: \"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.758790 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758745    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.758968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758812    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f9d878-fb9d-437c-b81d-defc288cc2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.758968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758848    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.758968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.758968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758901    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.758968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758941    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm99g\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-kube-api-access-fm99g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.758998    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759068    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759161    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759234    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759442 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759260    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759516 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759475    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759545    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.759643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.759623    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f9d878-fb9d-437c-b81d-defc288cc2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.761050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.761028    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.761622 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.761606    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.774559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.774534    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:28.782826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.782800    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.782924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.782894    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm99g\" (UniqueName: \"kubernetes.io/projected/96f9d878-fb9d-437c-b81d-defc288cc2be-kube-api-access-fm99g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-htjfk\" (UID: \"96f9d878-fb9d-437c-b81d-defc288cc2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:28.784667 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.784639    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:28.791274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.791247    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:28.809460 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:28.809218    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:29.099810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.099762    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"]
Nov 02 08:39:29.103143 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.103061    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"]
Nov 02 08:39:29.105599 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105566    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.105704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105627    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.105704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105622    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.105704 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105660    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.105870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105711    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.105870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.105760    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.155035 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.152602    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"]
Nov 02 08:39:29.159699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.155620    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"]
Nov 02 08:39:29.159699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.157555    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.159699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.157617    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.159699 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.157653    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.160106 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:39:29.160072    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650daaeb_7c93_4844_8659_6bebaa064303.slice/crio-112d3fbf141414536a5dd64ca306fc64456f3c390de0232c2dcf63e256a560ef WatchSource:0}: Error finding container 112d3fbf141414536a5dd64ca306fc64456f3c390de0232c2dcf63e256a560ef: Status 404 returned error can't find the container with id 112d3fbf141414536a5dd64ca306fc64456f3c390de0232c2dcf63e256a560ef
Nov 02 08:39:29.165949 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.165920    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.166041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.165978    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.166041 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.166027    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.711115    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" event={"ID":"96f9d878-fb9d-437c-b81d-defc288cc2be","Type":"ContainerStarted","Data":"9410123bdfce1b05ae697733ae3f8ee2e3e8de46a44c096076ce2bd32c785661"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.711230    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" event={"ID":"96f9d878-fb9d-437c-b81d-defc288cc2be","Type":"ContainerStarted","Data":"b9a9f730b17d625610bc986579f609b715c0b946c4f178199854294615e4b4bd"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.714611    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" event={"ID":"650daaeb-7c93-4844-8659-6bebaa064303","Type":"ContainerStarted","Data":"af4eff0c798062d66b1daeca232f6d196104245fd20bbac835eb76b4a578f283"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.714670    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" event={"ID":"650daaeb-7c93-4844-8659-6bebaa064303","Type":"ContainerStarted","Data":"112d3fbf141414536a5dd64ca306fc64456f3c390de0232c2dcf63e256a560ef"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.717388    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" event={"ID":"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a","Type":"ContainerStarted","Data":"09e7f6fd99ff12d94a4c595d5770f75ef416ab67594f57bfb2efd0163c977a46"}
Nov 02 08:39:29.720330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.717441    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" event={"ID":"a63a70fa-8feb-47e4-8cef-fd3b1b234b5a","Type":"ContainerStarted","Data":"4e98462b497627c796501755644420db93fa47780f5bac93f3d19ba57987d689"}
Nov 02 08:39:29.729332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.725815    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" event={"ID":"3629c3ef-7d97-4239-a095-71741d6885af","Type":"ContainerStarted","Data":"480616883ede2459dae301a164f7e5381896be9ee6af762adb000cc0aa98a838"}
Nov 02 08:39:29.729332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.725873    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" event={"ID":"3629c3ef-7d97-4239-a095-71741d6885af","Type":"ContainerStarted","Data":"d7029048ab20eb80b1c8a5d29d039384f4c107a3778531a7d304e540ffc518c3"}
Nov 02 08:39:29.774333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.773167    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podStartSLOduration=1.773136138 podStartE2EDuration="1.773136138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:29.771132896 +0000 UTC m=+4514.533569811" watchObservedRunningTime="2026-11-02 08:39:29.773136138 +0000 UTC m=+4514.535573046"
Nov 02 08:39:29.778330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.777965    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:29.787332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.785400    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:29.795337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.794620    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:29.812332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.811436    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:29.833332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.830767    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podStartSLOduration=1.830748125 podStartE2EDuration="1.830748125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:29.828937489 +0000 UTC m=+4514.591374401" watchObservedRunningTime="2026-11-02 08:39:29.830748125 +0000 UTC m=+4514.593185032"
Nov 02 08:39:29.939335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.938613    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podStartSLOduration=1.9385924829999999 podStartE2EDuration="1.938592483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:29.883808845 +0000 UTC m=+4514.646245752" watchObservedRunningTime="2026-11-02 08:39:29.938592483 +0000 UTC m=+4514.701029390"
Nov 02 08:39:29.939335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:29.939073    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podStartSLOduration=1.939059275 podStartE2EDuration="1.939059275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:29.936151371 +0000 UTC m=+4514.698588278" watchObservedRunningTime="2026-11-02 08:39:29.939059275 +0000 UTC m=+4514.701496182"
Nov 02 08:39:30.783488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.783444    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:30.784128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.784092    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:30.788927 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.788894    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:30.789124 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.789089    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:30.792700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.792671    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:30.792890 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.792866    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:30.811054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.811005    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:30.811374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:30.811339    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.790993    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.791055    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.791780    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.791821    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.792093    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:31.792336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.792122    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:31.823339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.810768    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:31.823339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:31.810824    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:32.776333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.775763    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:32.776333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.775826    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:32.792333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.788759    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:32.792333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.788820    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:32.792995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.792669    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:32.792995 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.792718    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:32.813334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.810731    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:32.813334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:32.810793    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:33.787335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.787149    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:33.787335 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.787219    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:33.791332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.787953    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:33.791332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.788002    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:33.797331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.792670    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:33.797331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.792714    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:33.811334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.810453    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:33.811334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:33.810677    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:34.779330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.778838    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:34.779330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.778917    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:34.787334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.786766    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:34.787334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.786840    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:34.802334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.801546    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:34.802334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.801605    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:34.814333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.811674    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:34.814333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:34.811733    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:35.775916 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.775881    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:35.776137 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.775942    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:35.785471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.785429    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:35.785651 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.785500    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:35.791849 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.791817    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:35.791987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.791872    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:35.809618 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.809578    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:35.810093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:35.809630    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:36.775295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.775250    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 08:39:36.775541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.775342    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 08:39:36.785695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.785593    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Nov 02 08:39:36.785695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.785665    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Nov 02 08:39:36.792071 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.792031    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 02 08:39:36.792259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.792100    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 02 08:39:36.810128 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.810089    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 02 08:39:36.810643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:36.810150    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 02 08:39:38.779609 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.779558    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:38.780252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.780207    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:38.787641 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.787597    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:38.787810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.787661    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:38.793288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.793256    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:38.793463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.793331    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:38.812936 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.812905    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:38.813108 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:38.812954    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:39.829348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.828416    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:39.829348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.828486    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:39.837330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.833412    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:39.837330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.833480    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:39.838903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.837594    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:39.838903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.837668    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:39.861332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.860411    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:39.861332 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:39.860482    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:40.875336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.874406    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:40.875336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.874473    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:40.880337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.879389    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:40.880337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.879441    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:40.884718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.884407    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:40.884718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.884467    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:40.910830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.910394    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:39:40.910830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:40.910451    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:39:41.466886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.454600    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:41.466886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.455971    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:41.466886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.456848    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9"
Nov 02 08:39:41.506496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.506462    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:41.507006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.506944    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:41.509807 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.509222    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28"
Nov 02 08:39:41.546593 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.546564    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:41.547374 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.547357    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:41.547918 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.547895    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7"
Nov 02 08:39:41.720957 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.720873    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:41.721235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.721213    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:41.721967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:41.721946    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk"
Nov 02 08:39:47.143565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.143519    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:47.144094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.143752    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" podUID="798484b6-1710-4d0f-80d9-ebf7a690c259" containerName="manager" containerID="cri-o://7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf" gracePeriod=30
Nov 02 08:39:47.399199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.399134    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:47.411195 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.411159    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\") pod \"798484b6-1710-4d0f-80d9-ebf7a690c259\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") "
Nov 02 08:39:47.411325 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.411298    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\") pod \"798484b6-1710-4d0f-80d9-ebf7a690c259\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") "
Nov 02 08:39:47.411386 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.411373    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\" (UniqueName: \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\") pod \"798484b6-1710-4d0f-80d9-ebf7a690c259\" (UID: \"798484b6-1710-4d0f-80d9-ebf7a690c259\") "
Nov 02 08:39:47.413288 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.413255    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert" pod "798484b6-1710-4d0f-80d9-ebf7a690c259" (UID: "798484b6-1710-4d0f-80d9-ebf7a690c259"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:47.413429 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.413394    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs" pod "798484b6-1710-4d0f-80d9-ebf7a690c259" (UID: "798484b6-1710-4d0f-80d9-ebf7a690c259"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:39:47.413528 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.413511    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx" pod "798484b6-1710-4d0f-80d9-ebf7a690c259" (UID: "798484b6-1710-4d0f-80d9-ebf7a690c259"). InnerVolumeSpecName "kube-api-access-kcvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:39:47.512821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.512778    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:47.512821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.512815    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvcx\" (UniqueName: \"kubernetes.io/projected/798484b6-1710-4d0f-80d9-ebf7a690c259-kube-api-access-kcvcx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:47.512821 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.512826    4515 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798484b6-1710-4d0f-80d9-ebf7a690c259-webhook-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:39:47.722255 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:47.722157    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:39:47.904235 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.904199    4515 generic.go:356] "Generic (PLEG): container finished" podID="798484b6-1710-4d0f-80d9-ebf7a690c259" containerID="7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf" exitCode=0
Nov 02 08:39:47.904426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.904248    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb"
Nov 02 08:39:47.904426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.904287    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" event={"ID":"798484b6-1710-4d0f-80d9-ebf7a690c259","Type":"ContainerDied","Data":"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"}
Nov 02 08:39:47.904426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.904354    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-hmrvb" event={"ID":"798484b6-1710-4d0f-80d9-ebf7a690c259","Type":"ContainerDied","Data":"eb1e3b0d6f74c797ada2e516aa3203b8821b2993edd46a62192cbf3b0550357b"}
Nov 02 08:39:47.904426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.904377    4515 scope.go:122] "RemoveContainer" containerID="7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"
Nov 02 08:39:47.914035 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.914016    4515 scope.go:122] "RemoveContainer" containerID="7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"
Nov 02 08:39:47.914325 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:39:47.914291    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf\": container with ID starting with 7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf not found: ID does not exist" containerID="7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"
Nov 02 08:39:47.914401 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.914330    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf"} err="failed to get container status \"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf\": rpc error: code = NotFound desc = could not find container \"7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf\": container with ID starting with 7c11e97aabcd22ecdc78e8cd1657baa90238b5fac83a93835ce6e426b34ef3bf not found: ID does not exist"
Nov 02 08:39:47.959681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.959651    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:47.972202 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:47.972175    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-hmrvb"]
Nov 02 08:39:49.133131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.133087    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-z74vw"]
Nov 02 08:39:49.133565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.133484    4515 cpu_manager.go:414] "containerMap: removing container" podUID="798484b6-1710-4d0f-80d9-ebf7a690c259" containerName="manager"
Nov 02 08:39:49.133565 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.133497    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798484b6-1710-4d0f-80d9-ebf7a690c259" containerName="manager"
Nov 02 08:39:49.133646 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.133583    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="798484b6-1710-4d0f-80d9-ebf7a690c259" containerName="manager"
Nov 02 08:39:49.138006 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.137987    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.153792 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.153750    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 08:39:49.166015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.165982    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 08:39:49.166015 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.165999    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xfsgv\""
Nov 02 08:39:49.166253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.166054    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 08:39:49.168068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.168045    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-z74vw"]
Nov 02 08:39:49.230660 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.230624    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-metrics-certs\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.230660 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.230666    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-webhook-cert\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.230958 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.230697    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzxf\" (UniqueName: \"kubernetes.io/projected/21eed327-37c2-4e36-8964-d9618c41cb58-kube-api-access-kvzxf\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.332330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.332264    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-metrics-certs\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.332330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.332328    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-webhook-cert\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.332543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.332365    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzxf\" (UniqueName: \"kubernetes.io/projected/21eed327-37c2-4e36-8964-d9618c41cb58-kube-api-access-kvzxf\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.334816 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.334785    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-webhook-cert\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.334951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.334861    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21eed327-37c2-4e36-8964-d9618c41cb58-metrics-certs\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.356390 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.356362    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzxf\" (UniqueName: \"kubernetes.io/projected/21eed327-37c2-4e36-8964-d9618c41cb58-kube-api-access-kvzxf\") pod \"maas-controller-5f6b5c786d-z74vw\" (UID: \"21eed327-37c2-4e36-8964-d9618c41cb58\") " pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.371610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.371581    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798484b6-1710-4d0f-80d9-ebf7a690c259" path="/var/lib/kubelet/pods/798484b6-1710-4d0f-80d9-ebf7a690c259/volumes"
Nov 02 08:39:49.450365 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.450263    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:49.752566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.752533    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f6b5c786d-z74vw"]
Nov 02 08:39:49.752973 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:39:49.752950    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21eed327_37c2_4e36_8964_d9618c41cb58.slice/crio-b8de13718be9c4dbc4a8f8c22af1fd8aa381a080825fb1f37854363cbc6dd1ed WatchSource:0}: Error finding container b8de13718be9c4dbc4a8f8c22af1fd8aa381a080825fb1f37854363cbc6dd1ed: Status 404 returned error can't find the container with id b8de13718be9c4dbc4a8f8c22af1fd8aa381a080825fb1f37854363cbc6dd1ed
Nov 02 08:39:49.915627 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:49.915596    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-z74vw" event={"ID":"21eed327-37c2-4e36-8964-d9618c41cb58","Type":"ContainerStarted","Data":"b8de13718be9c4dbc4a8f8c22af1fd8aa381a080825fb1f37854363cbc6dd1ed"}
Nov 02 08:39:50.921252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:50.921207    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f6b5c786d-z74vw" event={"ID":"21eed327-37c2-4e36-8964-d9618c41cb58","Type":"ContainerStarted","Data":"c6f70e06176b61ca2f7f886805ccc1c1fb30993f70bb59d1432b7d7a26c353a1"}
Nov 02 08:39:50.921683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:50.921386    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:39:50.980840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:39:50.980768    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f6b5c786d-z74vw" podStartSLOduration=1.548447465 podStartE2EDuration="1.98075335s" totalImagesPullingTime="432.305885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:50.978019481 +0000 UTC m=+4535.740456388" watchObservedRunningTime="2026-11-02 08:39:50.98075335 +0000 UTC m=+4535.743190256"
Nov 02 08:40:01.928448 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:01.928363    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f6b5c786d-z74vw"
Nov 02 08:40:10.502001 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:10.501957    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:40:12.770506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:12.770445    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:40:12.771207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:12.771167    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:40:17.721689 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:40:17.721647    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:40:26.218194 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:26.218155    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:40:32.534946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:32.534908    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:40:38.180334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:38.175508    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:40:42.813005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:40:42.812954    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:40:47.722494 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:40:47.722448    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:41:17.721962 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:41:17.721924    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:41:17.934923 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:17.934881    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:41:20.368717 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.368678    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:41:20.372255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.372238    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382182    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382250    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmr8\" (UniqueName: \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382284    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382377    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382438 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382428    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.382507 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.382457    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.402112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.402086    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-5b37c7ac-kserve-self-signed-certs\""
Nov 02 08:41:20.403811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.403789    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:41:20.482863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.482821    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.482863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.482867    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.482932    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.482976    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmr8\" (UniqueName: \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.483005    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483210 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.483083    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.483366    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.483388    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.483497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.483429    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.485088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.485072    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.485643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.485621    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.506991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.506963    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmr8\" (UniqueName: \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\") pod \"e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.701907 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.701808    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:20.747645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.747606    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:41:20.753116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.751484    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.770162 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.770108    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-5b37c7ac-epp-sa-dockercfg-wzkhn\""
Nov 02 08:41:20.781865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.781834    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:41:20.786803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.786772    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.786933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.786814    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjgk\" (UniqueName: \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.887880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.887843    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.888057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.887887    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjgk\" (UniqueName: \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.890362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.890338    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:20.911570 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:20.911540    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjgk\" (UniqueName: \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\") pod \"e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") " pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:21.004637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.004598    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:41:21.005880 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:41:21.005836    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ca9fc13_3ab5_443f_98cb_23f8ea0d4c2c.slice/crio-4dbf203ce28aaa5293f08fd45d6de0233c8c10e5411e823541ee5b0a19ded296 WatchSource:0}: Error finding container 4dbf203ce28aaa5293f08fd45d6de0233c8c10e5411e823541ee5b0a19ded296: Status 404 returned error can't find the container with id 4dbf203ce28aaa5293f08fd45d6de0233c8c10e5411e823541ee5b0a19ded296
Nov 02 08:41:21.076362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.076334    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:21.376230 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:41:21.376200    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1334d9_2a60_4c42_a794_55fa77c5becd.slice/crio-c17d6b07411b4bad743805f734ed8aa01300c99e6a6ec0b98a7bc65e2ac3f173 WatchSource:0}: Error finding container c17d6b07411b4bad743805f734ed8aa01300c99e6a6ec0b98a7bc65e2ac3f173: Status 404 returned error can't find the container with id c17d6b07411b4bad743805f734ed8aa01300c99e6a6ec0b98a7bc65e2ac3f173
Nov 02 08:41:21.376603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.376252    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:41:21.403873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.403825    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" event={"ID":"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c","Type":"ContainerStarted","Data":"2afe3fa45fe69d534fbfb479e6ccdba52769b344b90872eccda76885acfae882"}
Nov 02 08:41:21.404057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.403881    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" event={"ID":"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c","Type":"ContainerStarted","Data":"4dbf203ce28aaa5293f08fd45d6de0233c8c10e5411e823541ee5b0a19ded296"}
Nov 02 08:41:21.405138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.405108    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" event={"ID":"db1334d9-2a60-4c42-a794-55fa77c5becd","Type":"ContainerStarted","Data":"c17d6b07411b4bad743805f734ed8aa01300c99e6a6ec0b98a7bc65e2ac3f173"}
Nov 02 08:41:21.447225 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.447173    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" podStartSLOduration=1.447160873 podStartE2EDuration="1.447160873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:21.446167435 +0000 UTC m=+4626.208604340" watchObservedRunningTime="2026-11-02 08:41:21.447160873 +0000 UTC m=+4626.209597772"
Nov 02 08:41:21.702544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:21.702461    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:23.241336 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:23.241288    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:41:23.415544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:23.415508    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" event={"ID":"db1334d9-2a60-4c42-a794-55fa77c5becd","Type":"ContainerStarted","Data":"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"}
Nov 02 08:41:23.461324 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:23.461260    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podStartSLOduration=2.092946526 podStartE2EDuration="3.461247716s" totalImagesPullingTime="1.36830119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:41:23.459156405 +0000 UTC m=+4628.221593311" watchObservedRunningTime="2026-11-02 08:41:23.461247716 +0000 UTC m=+4628.223684622"
Nov 02 08:41:24.422643 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:24.422603    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:41:30.702700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:30.702616    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:30.715530 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:30.715501    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:31.484444 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:31.484412    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:41:47.722766 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:41:47.722728    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:41:55.437920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:41:55.437886    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:42:02.306345 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:02.306295    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:42:17.723383 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:42:17.723345    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:42:21.316623 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:21.316584    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:42:21.606816 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:21.606736    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:42:21.615685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:21.615657    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:42:25.437872 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:25.437822    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:42:27.501461 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.501418    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:42:27.502200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.502166    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:42:27.549701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.549559    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:42:27.549701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.549622    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:42:27.606334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.597734    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:42:27.606334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.598162    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:42:27.768396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.766430    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:42:27.768396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:27.766499    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:42:28.646105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:28.646070    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:42:35.438764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:35.438714    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:42:36.628274 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:36.628235    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" containerID="cri-o://23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a" gracePeriod=45
Nov 02 08:42:36.635487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:36.635451    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" podUID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" containerName="main" containerID="cri-o://2afe3fa45fe69d534fbfb479e6ccdba52769b344b90872eccda76885acfae882" gracePeriod=45
Nov 02 08:42:36.846182 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:36.846152    4515 generic.go:356] "Generic (PLEG): container finished" podID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" containerID="2afe3fa45fe69d534fbfb479e6ccdba52769b344b90872eccda76885acfae882" exitCode=0
Nov 02 08:42:36.846426 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:36.846232    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" event={"ID":"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c","Type":"ContainerDied","Data":"2afe3fa45fe69d534fbfb479e6ccdba52769b344b90872eccda76885acfae882"}
Nov 02 08:42:36.904680 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:36.904656    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:42:37.000057 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000018    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000129    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000166    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000195    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000240    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000291    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\" (UniqueName: \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\") pod \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\" (UID: \"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c\") "
Nov 02 08:42:37.000543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000495    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:42:37.000543 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000515    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:42:37.000738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000550    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:42:37.000738 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000730    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.000858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000749    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.000858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.000762    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.002489 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.002456    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:42:37.002610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.002513    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "kube-api-access-phmr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:42:37.002610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.002519    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm" pod "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" (UID: "4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:42:37.101450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.101408    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmr8\" (UniqueName: \"kubernetes.io/projected/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-kube-api-access-phmr8\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.101671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.101455    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.101671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.101473    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:42:37.852758 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.852726    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"
Nov 02 08:42:37.853251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.852731    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5" event={"ID":"4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c","Type":"ContainerDied","Data":"4dbf203ce28aaa5293f08fd45d6de0233c8c10e5411e823541ee5b0a19ded296"}
Nov 02 08:42:37.853251 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.852857    4515 scope.go:122] "RemoveContainer" containerID="2afe3fa45fe69d534fbfb479e6ccdba52769b344b90872eccda76885acfae882"
Nov 02 08:42:37.904701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.904665    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:42:37.917174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:37.917143    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-78f764db84-2rjx5"]
Nov 02 08:42:39.371154 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:39.371116    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" path="/var/lib/kubelet/pods/4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c/volumes"
Nov 02 08:42:45.437445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:45.437401    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:42:45.437850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:45.437508    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:42:45.735820 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:45.735727    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:42:47.722095 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:42:47.722054    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:42:55.437573 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:42:55.437531    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:43:05.437823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:05.437739    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:43:06.901793 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.901769    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:43:06.983914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.983858    4515 generic.go:356] "Generic (PLEG): container finished" podID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerID="23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a" exitCode=0
Nov 02 08:43:06.984212 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.984113    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"
Nov 02 08:43:06.984379 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.984066    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" event={"ID":"db1334d9-2a60-4c42-a794-55fa77c5becd","Type":"ContainerDied","Data":"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"}
Nov 02 08:43:06.984449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.984434    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh" event={"ID":"db1334d9-2a60-4c42-a794-55fa77c5becd","Type":"ContainerDied","Data":"c17d6b07411b4bad743805f734ed8aa01300c99e6a6ec0b98a7bc65e2ac3f173"}
Nov 02 08:43:06.984490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.984469    4515 scope.go:122] "RemoveContainer" containerID="23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"
Nov 02 08:43:06.992020 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.991994    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\") pod \"db1334d9-2a60-4c42-a794-55fa77c5becd\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") "
Nov 02 08:43:06.992144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.992072    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\" (UniqueName: \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\") pod \"db1334d9-2a60-4c42-a794-55fa77c5becd\" (UID: \"db1334d9-2a60-4c42-a794-55fa77c5becd\") "
Nov 02 08:43:06.994174 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.994134    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs" pod "db1334d9-2a60-4c42-a794-55fa77c5becd" (UID: "db1334d9-2a60-4c42-a794-55fa77c5becd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:43:06.994281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.994181    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk" pod "db1334d9-2a60-4c42-a794-55fa77c5becd" (UID: "db1334d9-2a60-4c42-a794-55fa77c5becd"). InnerVolumeSpecName "kube-api-access-5kjgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:43:06.994746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.994723    4515 scope.go:122] "RemoveContainer" containerID="23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"
Nov 02 08:43:06.996591 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:43:06.996566    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a\": container with ID starting with 23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a not found: ID does not exist" containerID="23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"
Nov 02 08:43:06.996687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:06.996600    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a"} err="failed to get container status \"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a\": rpc error: code = NotFound desc = could not find container \"23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a\": container with ID starting with 23dad581d512793253661ebb716bc3d5f50eab8130aa279542c8ad55bf32962a not found: ID does not exist"
Nov 02 08:43:07.093560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:07.093531    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1334d9-2a60-4c42-a794-55fa77c5becd-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:43:07.093560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:07.093557    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjgk\" (UniqueName: \"kubernetes.io/projected/db1334d9-2a60-4c42-a794-55fa77c5becd-kube-api-access-5kjgk\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:43:07.332882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:07.332845    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:43:07.339519 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:07.339491    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-5b37c7ac-kserve-router-scheduler-56444cbb55-fdrmh"]
Nov 02 08:43:07.371553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:07.371517    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" path="/var/lib/kubelet/pods/db1334d9-2a60-4c42-a794-55fa77c5becd/volumes"
Nov 02 08:43:09.232337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:09.232276    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:43:17.722655 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:43:17.722618    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:43:26.928677 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:26.928636    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:43:47.724109 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:43:47.724069    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:43:51.145717 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:43:51.145684    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:44:02.146924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:02.146888    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:44:17.722181 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:44:17.722138    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:44:31.861457 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861373    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:31.861886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861769    4515 cpu_manager.go:414] "containerMap: removing container" podUID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" containerName="main"
Nov 02 08:44:31.861886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861781    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" containerName="main"
Nov 02 08:44:31.861886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861813    4515 cpu_manager.go:414] "containerMap: removing container" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main"
Nov 02 08:44:31.861886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861818    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main"
Nov 02 08:44:31.861886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861884    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1334d9-2a60-4c42-a794-55fa77c5becd" containerName="main"
Nov 02 08:44:31.862066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.861894    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca9fc13-3ab5-443f-98cb-23f8ea0d4c2c" containerName="main"
Nov 02 08:44:31.865003 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.864981    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:31.880349 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.880328    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1b623b-serving-cert\""
Nov 02 08:44:31.894947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:31.894920    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:32.030991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.030950    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.031184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.031128    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4x4c\" (UniqueName: \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.031184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.031162    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.031346 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.031201    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.132207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.132117    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4x4c\" (UniqueName: \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.132207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.132160    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.132207 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.132204    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.132484 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.132251    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.132997 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.132975    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.134851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.134818    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.134975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.134857    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.159950 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.159913    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4x4c\" (UniqueName: \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.177777 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.177747    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:32.314996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.314955    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:32.318504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.318488    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.335853 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.335826    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-43c72653-dockercfg-m8bq4\""
Nov 02 08:44:32.336778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.336760    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1b623b\""
Nov 02 08:44:32.341839 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.341816    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:32.414955 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.414872    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:32.418277 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.418259    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.436879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.436851    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.437039 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.437018    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2t5\" (UniqueName: \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.446302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.446271    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:32.534508 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.532341    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:32.540273 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.540226    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2t5\" (UniqueName: \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.540576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.540550    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.540675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.540624    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.540740 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.540687    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cshm\" (UniqueName: \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.541387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.541367    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.550128 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:44:32.550104    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff3fa41_ba18_4ec6_b4b5_8bd9a529da42.slice/crio-9c6d61e027e777234b50de10b0a6dbd32706080a3c25e10179422d5de9bb859b WatchSource:0}: Error finding container 9c6d61e027e777234b50de10b0a6dbd32706080a3c25e10179422d5de9bb859b: Status 404 returned error can't find the container with id 9c6d61e027e777234b50de10b0a6dbd32706080a3c25e10179422d5de9bb859b
Nov 02 08:44:32.551437 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.551422    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 08:44:32.586049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.586012    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2t5\" (UniqueName: \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.631838 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.631803    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:32.641531 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.641493    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.641705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.641590    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cshm\" (UniqueName: \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.642284 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.642260    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.673356 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.673256    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cshm\" (UniqueName: \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.730719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.730677    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:32.953545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:32.953514    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:32.954010 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:44:32.953990    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81fa4ed_6ec7_484d_a7cd_e62a1cb7b4d8.slice/crio-3058fc56a5335b148860b5e3d44fdd395907d69d84e72d387fc4fbfefa106ca5 WatchSource:0}: Error finding container 3058fc56a5335b148860b5e3d44fdd395907d69d84e72d387fc4fbfefa106ca5: Status 404 returned error can't find the container with id 3058fc56a5335b148860b5e3d44fdd395907d69d84e72d387fc4fbfefa106ca5
Nov 02 08:44:33.048340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.048249    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:33.051342 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:44:33.050589    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d481f3c_08c6_4feb_be71_a6d059da1d79.slice/crio-9400f065e395c69a66eb652bae9d073ed11bcc1618563c00825fc212a9f2c9f7 WatchSource:0}: Error finding container 9400f065e395c69a66eb652bae9d073ed11bcc1618563c00825fc212a9f2c9f7: Status 404 returned error can't find the container with id 9400f065e395c69a66eb652bae9d073ed11bcc1618563c00825fc212a9f2c9f7
Nov 02 08:44:33.374650 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.374612    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" event={"ID":"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8","Type":"ContainerStarted","Data":"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"}
Nov 02 08:44:33.374866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.374669    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" event={"ID":"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8","Type":"ContainerStarted","Data":"3058fc56a5335b148860b5e3d44fdd395907d69d84e72d387fc4fbfefa106ca5"}
Nov 02 08:44:33.374866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.374699    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:33.376254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.376226    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" event={"ID":"5d481f3c-08c6-4feb-be71-a6d059da1d79","Type":"ContainerStarted","Data":"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"}
Nov 02 08:44:33.376432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.376269    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" event={"ID":"5d481f3c-08c6-4feb-be71-a6d059da1d79","Type":"ContainerStarted","Data":"9400f065e395c69a66eb652bae9d073ed11bcc1618563c00825fc212a9f2c9f7"}
Nov 02 08:44:33.376432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.376331    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:33.377888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.377860    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" event={"ID":"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42","Type":"ContainerStarted","Data":"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"}
Nov 02 08:44:33.378060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.377908    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" event={"ID":"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42","Type":"ContainerStarted","Data":"9c6d61e027e777234b50de10b0a6dbd32706080a3c25e10179422d5de9bb859b"}
Nov 02 08:44:33.378060 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.377997    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:33.427915 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.427863    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" podStartSLOduration=1.427849188 podStartE2EDuration="1.427849188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:33.427426149 +0000 UTC m=+4818.189863055" watchObservedRunningTime="2026-11-02 08:44:33.427849188 +0000 UTC m=+4818.190286121"
Nov 02 08:44:33.473566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.473511    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" podStartSLOduration=1.473496918 podStartE2EDuration="1.473496918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:33.472971344 +0000 UTC m=+4818.235408250" watchObservedRunningTime="2026-11-02 08:44:33.473496918 +0000 UTC m=+4818.235933821"
Nov 02 08:44:33.520796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:33.520747    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" podStartSLOduration=2.119496148 podStartE2EDuration="2.520734627s" totalImagesPullingTime="401.238479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:33.519589123 +0000 UTC m=+4818.282026029" watchObservedRunningTime="2026-11-02 08:44:33.520734627 +0000 UTC m=+4818.283171534"
Nov 02 08:44:37.650879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:37.650843    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:44:39.389396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:39.389368    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:44.387037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:44.387000    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:44.387503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:44.387157    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:47.722660 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:44:47.722618    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:44:51.832269 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:51.832232    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:51.866278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:51.866241    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:53.907933 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:53.907896    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:53.908620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:53.908120    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" podUID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" containerName="maas-api" containerID="cri-o://94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8" gracePeriod=30
Nov 02 08:44:54.160211 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.160150    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:54.265144 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265087    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\") pod \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") "
Nov 02 08:44:54.265361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265155    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\" (UniqueName: \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\") pod \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") "
Nov 02 08:44:54.265361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265199    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\") pod \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") "
Nov 02 08:44:54.265361 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265247    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\") pod \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\" (UID: \"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42\") "
Nov 02 08:44:54.265664 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265634    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca" pod "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" (UID: "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:44:54.265811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.265741    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:54.267319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.267283    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls" pod "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" (UID: "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:44:54.267449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.267423    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c" pod "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" (UID: "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42"). InnerVolumeSpecName "kube-api-access-h4x4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:44:54.267449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.267430    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs" pod "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" (UID: "8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:44:54.366601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.366561    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:54.366601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.366596    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4x4c\" (UniqueName: \"kubernetes.io/projected/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-kube-api-access-h4x4c\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:54.366601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.366608    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:54.477116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.477018    4515 generic.go:356] "Generic (PLEG): container finished" podID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" containerID="94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8" exitCode=0
Nov 02 08:44:54.477116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.477074    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"
Nov 02 08:44:54.477116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.477082    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" event={"ID":"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42","Type":"ContainerDied","Data":"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"}
Nov 02 08:44:54.477413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.477142    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn" event={"ID":"8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42","Type":"ContainerDied","Data":"9c6d61e027e777234b50de10b0a6dbd32706080a3c25e10179422d5de9bb859b"}
Nov 02 08:44:54.477413 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.477174    4515 scope.go:122] "RemoveContainer" containerID="94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"
Nov 02 08:44:54.486232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.486213    4515 scope.go:122] "RemoveContainer" containerID="94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"
Nov 02 08:44:54.486529 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:44:54.486512    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8\": container with ID starting with 94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8 not found: ID does not exist" containerID="94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"
Nov 02 08:44:54.486601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.486536    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8"} err="failed to get container status \"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8\": rpc error: code = NotFound desc = could not find container \"94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8\": container with ID starting with 94bbd134c89b7600a269722313ed11646baa4a7e1465aceebae80e2039eb43b8 not found: ID does not exist"
Nov 02 08:44:54.550655 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.550620    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:54.568352 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:54.568293    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-8j2kn"]
Nov 02 08:44:55.373354 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:55.373296    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" path="/var/lib/kubelet/pods/8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42/volumes"
Nov 02 08:44:56.541517 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:56.541481    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:44:56.833410 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:56.833296    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" podUID="5d481f3c-08c6-4feb-be71-a6d059da1d79" containerName="payload-processing" containerID="cri-o://234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15" gracePeriod=25
Nov 02 08:44:56.866913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:56.866846    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" podUID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" containerName="payload-pre-processing" containerID="cri-o://2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f" gracePeriod=25
Nov 02 08:44:57.115867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.115841    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:57.140864 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.140839    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:57.195841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.195802    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\") pod \"5d481f3c-08c6-4feb-be71-a6d059da1d79\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") "
Nov 02 08:44:57.196050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.195959    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\" (UniqueName: \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\") pod \"5d481f3c-08c6-4feb-be71-a6d059da1d79\" (UID: \"5d481f3c-08c6-4feb-be71-a6d059da1d79\") "
Nov 02 08:44:57.196217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.196193    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume" pod "5d481f3c-08c6-4feb-be71-a6d059da1d79" (UID: "5d481f3c-08c6-4feb-be71-a6d059da1d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:44:57.198037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.198017    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm" pod "5d481f3c-08c6-4feb-be71-a6d059da1d79" (UID: "5d481f3c-08c6-4feb-be71-a6d059da1d79"). InnerVolumeSpecName "kube-api-access-7cshm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:44:57.296823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.296784    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\") pod \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") "
Nov 02 08:44:57.297012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.296836    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\" (UniqueName: \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\") pod \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\" (UID: \"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8\") "
Nov 02 08:44:57.297105 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.297086    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d481f3c-08c6-4feb-be71-a6d059da1d79-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:57.297150 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.297108    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cshm\" (UniqueName: \"kubernetes.io/projected/5d481f3c-08c6-4feb-be71-a6d059da1d79-kube-api-access-7cshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:57.297278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.297252    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume" pod "e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" (UID: "e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:44:57.298968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.298950    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5" pod "e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" (UID: "e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8"). InnerVolumeSpecName "kube-api-access-7b2t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:44:57.398051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.398022    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:57.398051 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.398049    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b2t5\" (UniqueName: \"kubernetes.io/projected/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8-kube-api-access-7b2t5\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:44:57.492168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.492128    4515 generic.go:356] "Generic (PLEG): container finished" podID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" containerID="2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f" exitCode=0
Nov 02 08:44:57.492376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.492180    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"
Nov 02 08:44:57.492376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.492205    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" event={"ID":"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8","Type":"ContainerDied","Data":"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"}
Nov 02 08:44:57.492376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.492258    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58" event={"ID":"e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8","Type":"ContainerDied","Data":"3058fc56a5335b148860b5e3d44fdd395907d69d84e72d387fc4fbfefa106ca5"}
Nov 02 08:44:57.492376 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.492283    4515 scope.go:122] "RemoveContainer" containerID="2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"
Nov 02 08:44:57.493809 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.493789    4515 generic.go:356] "Generic (PLEG): container finished" podID="5d481f3c-08c6-4feb-be71-a6d059da1d79" containerID="234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15" exitCode=0
Nov 02 08:44:57.493914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.493833    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"
Nov 02 08:44:57.493914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.493841    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" event={"ID":"5d481f3c-08c6-4feb-be71-a6d059da1d79","Type":"ContainerDied","Data":"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"}
Nov 02 08:44:57.493914 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.493882    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7" event={"ID":"5d481f3c-08c6-4feb-be71-a6d059da1d79","Type":"ContainerDied","Data":"9400f065e395c69a66eb652bae9d073ed11bcc1618563c00825fc212a9f2c9f7"}
Nov 02 08:44:57.501818 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.501802    4515 scope.go:122] "RemoveContainer" containerID="2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"
Nov 02 08:44:57.502179 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:44:57.502149    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f\": container with ID starting with 2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f not found: ID does not exist" containerID="2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"
Nov 02 08:44:57.502285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.502191    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f"} err="failed to get container status \"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f\": rpc error: code = NotFound desc = could not find container \"2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f\": container with ID starting with 2ef7969d49d157641ab22c1dbcb6d415fad19c854c6dcca78e3409bda1a2d90f not found: ID does not exist"
Nov 02 08:44:57.502285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.502208    4515 scope.go:122] "RemoveContainer" containerID="234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"
Nov 02 08:44:57.510137 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.510109    4515 scope.go:122] "RemoveContainer" containerID="234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"
Nov 02 08:44:57.510412 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:44:57.510397    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15\": container with ID starting with 234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15 not found: ID does not exist" containerID="234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"
Nov 02 08:44:57.510477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.510420    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15"} err="failed to get container status \"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15\": rpc error: code = NotFound desc = could not find container \"234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15\": container with ID starting with 234a567538de24e8d6d2347a97789fc3940bfa6871e442cba6b7168eb9a52e15 not found: ID does not exist"
Nov 02 08:44:57.560614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.560583    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:57.607358 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.607326    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7zxsj7"]
Nov 02 08:44:57.658034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.657942    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:57.662888 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:57.662858    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8mfm58"]
Nov 02 08:44:59.371034 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:59.370997    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d481f3c-08c6-4feb-be71-a6d059da1d79" path="/var/lib/kubelet/pods/5d481f3c-08c6-4feb-be71-a6d059da1d79/volumes"
Nov 02 08:44:59.371630 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:44:59.371607    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" path="/var/lib/kubelet/pods/e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8/volumes"
Nov 02 08:45:01.083722 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.083686    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:45:01.084292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084272    4515 cpu_manager.go:414] "containerMap: removing container" podUID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" containerName="payload-pre-processing"
Nov 02 08:45:01.084368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084296    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" containerName="payload-pre-processing"
Nov 02 08:45:01.084368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084333    4515 cpu_manager.go:414] "containerMap: removing container" podUID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" containerName="maas-api"
Nov 02 08:45:01.084368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084341    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" containerName="maas-api"
Nov 02 08:45:01.084473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084374    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5d481f3c-08c6-4feb-be71-a6d059da1d79" containerName="payload-processing"
Nov 02 08:45:01.084473 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084383    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d481f3c-08c6-4feb-be71-a6d059da1d79" containerName="payload-processing"
Nov 02 08:45:01.084538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084484    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81fa4ed-6ec7-484d-a7cd-e62a1cb7b4d8" containerName="payload-pre-processing"
Nov 02 08:45:01.084538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084495    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff3fa41-ba18-4ec6-b4b5-8bd9a529da42" containerName="maas-api"
Nov 02 08:45:01.084538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.084510    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d481f3c-08c6-4feb-be71-a6d059da1d79" containerName="payload-processing"
Nov 02 08:45:01.088125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.088102    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.107544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.107515    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:45:01.118151 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.118125    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e7cd98d7-kserve-self-signed-certs\""
Nov 02 08:45:01.234205 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234172    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.234411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234219    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.234411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234394    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69xw\" (UniqueName: \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.234499 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234440    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.234499 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234489    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.234612 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.234516    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.335867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335775    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335880    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69xw\" (UniqueName: \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335918    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335939    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335969    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336049 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.335997    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.336275    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.336290    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.336495 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.336422    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.338357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.338336    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.338566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.338550    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.359589 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.359555    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69xw\" (UniqueName: \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\") pod \"e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") " pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.407449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.407408    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:01.711045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:01.711013    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:45:01.712081 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:45:01.712052    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e46f254_6527_40f3_bc5c_3fea46ffef25.slice/crio-f370ffd3bd063e3f1ce526f8fdd8e1af8316e8a5055286e8ff1ae8700ce4dbfd WatchSource:0}: Error finding container f370ffd3bd063e3f1ce526f8fdd8e1af8316e8a5055286e8ff1ae8700ce4dbfd: Status 404 returned error can't find the container with id f370ffd3bd063e3f1ce526f8fdd8e1af8316e8a5055286e8ff1ae8700ce4dbfd
Nov 02 08:45:02.520873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:02.520840    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" event={"ID":"6e46f254-6527-40f3-bc5c-3fea46ffef25","Type":"ContainerStarted","Data":"f32a58865e563b426e92826ab7efab7279569821d0fc051611202d3d31f3a519"}
Nov 02 08:45:02.521351 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:02.520881    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" event={"ID":"6e46f254-6527-40f3-bc5c-3fea46ffef25","Type":"ContainerStarted","Data":"f370ffd3bd063e3f1ce526f8fdd8e1af8316e8a5055286e8ff1ae8700ce4dbfd"}
Nov 02 08:45:02.566774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:02.566717    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" podStartSLOduration=1.566702579 podStartE2EDuration="1.566702579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:02.565830971 +0000 UTC m=+4847.328267859" watchObservedRunningTime="2026-11-02 08:45:02.566702579 +0000 UTC m=+4847.329139484"
Nov 02 08:45:03.407925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:03.407885    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:03.617387 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:03.617358    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:45:11.408267 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:11.408229    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:11.420822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:11.420791    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:11.648505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:11.648472    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:45:16.348955 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.348913    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:45:16.357947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.357918    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.371448 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.371420    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:45:16.382477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.382448    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e7cd98d7-kserve-self-signed-certs\""
Nov 02 08:45:16.509101 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509061    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.509337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509110    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.509337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509134    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.509337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509241    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8lp\" (UniqueName: \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.509337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509262    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.509537 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.509388    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610775 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610694    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610781    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610807    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610823    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610849    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8lp\" (UniqueName: \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.610926 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.610872    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.611200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.611169    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.611244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.611182    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.611244 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.611222    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.613255 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.613231    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.613491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.613473    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.634514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.634478    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8lp\" (UniqueName: \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\") pod \"e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") " pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.672045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.672013    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:16.969880 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:45:16.969849    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6c0a158_1bce_4737_a8bc_42db65560cbb.slice/crio-0b4a2e01b10ed1132b2d9f6aaf53396ae7fb792f85a22ce734795751ec48b387 WatchSource:0}: Error finding container 0b4a2e01b10ed1132b2d9f6aaf53396ae7fb792f85a22ce734795751ec48b387: Status 404 returned error can't find the container with id 0b4a2e01b10ed1132b2d9f6aaf53396ae7fb792f85a22ce734795751ec48b387
Nov 02 08:45:16.970024 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:16.969998    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:45:17.665259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:17.665209    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" event={"ID":"d6c0a158-1bce-4737-a8bc-42db65560cbb","Type":"ContainerStarted","Data":"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"}
Nov 02 08:45:17.665259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:17.665261    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" event={"ID":"d6c0a158-1bce-4737-a8bc-42db65560cbb","Type":"ContainerStarted","Data":"0b4a2e01b10ed1132b2d9f6aaf53396ae7fb792f85a22ce734795751ec48b387"}
Nov 02 08:45:17.672500 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:17.672483    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:17.723521 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:45:17.723480    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:45:18.812441 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:18.812386    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" podStartSLOduration=2.812369801 podStartE2EDuration="2.812369801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:17.71174888 +0000 UTC m=+4862.474185785" watchObservedRunningTime="2026-11-02 08:45:18.812369801 +0000 UTC m=+4863.574806707"
Nov 02 08:45:18.813463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:18.813442    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:45:26.672855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:26.672813    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:26.685475 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:26.685442    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:26.751261 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:26.751231    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:45:34.936485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:34.936450    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:45:44.742693 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:44.742660    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:45:46.141061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:46.141025    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:45:46.930116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:46.930083    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:45:47.722715 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:45:47.722681    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:45:53.637334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:53.599693    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:45:59.764140 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:59.764041    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" podUID="6e46f254-6527-40f3-bc5c-3fea46ffef25" containerName="main" containerID="cri-o://f32a58865e563b426e92826ab7efab7279569821d0fc051611202d3d31f3a519" gracePeriod=45
Nov 02 08:45:59.941409 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:59.941374    4515 generic.go:356] "Generic (PLEG): container finished" podID="6e46f254-6527-40f3-bc5c-3fea46ffef25" containerID="f32a58865e563b426e92826ab7efab7279569821d0fc051611202d3d31f3a519" exitCode=0
Nov 02 08:45:59.941560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:45:59.941422    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" event={"ID":"6e46f254-6527-40f3-bc5c-3fea46ffef25","Type":"ContainerDied","Data":"f32a58865e563b426e92826ab7efab7279569821d0fc051611202d3d31f3a519"}
Nov 02 08:46:00.021892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.021825    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:46:00.076012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.075980    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076024    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076075    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076134    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076270    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\" (UniqueName: \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076321    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\") pod \"6e46f254-6527-40f3-bc5c-3fea46ffef25\" (UID: \"6e46f254-6527-40f3-bc5c-3fea46ffef25\") "
Nov 02 08:46:00.076431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076385    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:00.076431 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076402    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:00.076685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076671    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.076733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076676    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:00.076733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.076691    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.078191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.078165    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:00.078446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.078423    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "kube-api-access-h69xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:46:00.078773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.078753    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs" pod "6e46f254-6527-40f3-bc5c-3fea46ffef25" (UID: "6e46f254-6527-40f3-bc5c-3fea46ffef25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:46:00.177741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.177694    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.177741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.177732    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e46f254-6527-40f3-bc5c-3fea46ffef25-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.177741 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.177745    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e46f254-6527-40f3-bc5c-3fea46ffef25-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.178000 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.177758    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69xw\" (UniqueName: \"kubernetes.io/projected/6e46f254-6527-40f3-bc5c-3fea46ffef25-kube-api-access-h69xw\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:00.948066 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.948031    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw" event={"ID":"6e46f254-6527-40f3-bc5c-3fea46ffef25","Type":"ContainerDied","Data":"f370ffd3bd063e3f1ce526f8fdd8e1af8316e8a5055286e8ff1ae8700ce4dbfd"}
Nov 02 08:46:00.948541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.948073    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"
Nov 02 08:46:00.948541 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:00.948083    4515 scope.go:122] "RemoveContainer" containerID="f32a58865e563b426e92826ab7efab7279569821d0fc051611202d3d31f3a519"
Nov 02 08:46:01.000694 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.000657    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:46:01.012934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.012902    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e7cd98d7-kserve-d84c5cb65-gvqhw"]
Nov 02 08:46:01.110262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.110224    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:46:01.161282 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.161232    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" podUID="d6c0a158-1bce-4737-a8bc-42db65560cbb" containerName="main" containerID="cri-o://d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63" gracePeriod=45
Nov 02 08:46:01.372824 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.372795    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e46f254-6527-40f3-bc5c-3fea46ffef25" path="/var/lib/kubelet/pods/6e46f254-6527-40f3-bc5c-3fea46ffef25/volumes"
Nov 02 08:46:01.411719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.411691    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:46:01.490920 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.490887    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.490982    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491025    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491108    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491175    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491213    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\" (UniqueName: \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\") pod \"d6c0a158-1bce-4737-a8bc-42db65560cbb\" (UID: \"d6c0a158-1bce-4737-a8bc-42db65560cbb\") "
Nov 02 08:46:01.491348 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491213    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:01.492398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.491624    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.492398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.492269    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:01.492398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.492366    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:01.493753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.493718    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:46:01.495171 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.495148    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "kube-api-access-2r8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:46:01.495511 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.495485    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs" pod "d6c0a158-1bce-4737-a8bc-42db65560cbb" (UID: "d6c0a158-1bce-4737-a8bc-42db65560cbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:46:01.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.592511    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8lp\" (UniqueName: \"kubernetes.io/projected/d6c0a158-1bce-4737-a8bc-42db65560cbb-kube-api-access-2r8lp\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.592541    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.592551    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.592560    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c0a158-1bce-4737-a8bc-42db65560cbb-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.592588 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.592568    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c0a158-1bce-4737-a8bc-42db65560cbb-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:01.953709 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.953622    4515 generic.go:356] "Generic (PLEG): container finished" podID="d6c0a158-1bce-4737-a8bc-42db65560cbb" containerID="d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63" exitCode=0
Nov 02 08:46:01.953709 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.953672    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"
Nov 02 08:46:01.954208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.953703    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" event={"ID":"d6c0a158-1bce-4737-a8bc-42db65560cbb","Type":"ContainerDied","Data":"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"}
Nov 02 08:46:01.954208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.953751    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b" event={"ID":"d6c0a158-1bce-4737-a8bc-42db65560cbb","Type":"ContainerDied","Data":"0b4a2e01b10ed1132b2d9f6aaf53396ae7fb792f85a22ce734795751ec48b387"}
Nov 02 08:46:01.954208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.953769    4515 scope.go:122] "RemoveContainer" containerID="d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"
Nov 02 08:46:01.962994 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.962973    4515 scope.go:122] "RemoveContainer" containerID="d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"
Nov 02 08:46:01.963316 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:46:01.963284    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63\": container with ID starting with d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63 not found: ID does not exist" containerID="d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"
Nov 02 08:46:01.963375 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:01.963341    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63"} err="failed to get container status \"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63\": rpc error: code = NotFound desc = could not find container \"d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63\": container with ID starting with d87b2f9e3fdbdd76351bde388ec57cbd9d93d1cd649e989bb684d1c623f8fc63 not found: ID does not exist"
Nov 02 08:46:02.005987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:02.005950    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:46:02.010669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:02.010639    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e7cd98d7-kserve-67f7cd47c7-h8m6b"]
Nov 02 08:46:03.371897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:03.371850    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c0a158-1bce-4737-a8bc-42db65560cbb" path="/var/lib/kubelet/pods/d6c0a158-1bce-4737-a8bc-42db65560cbb/volumes"
Nov 02 08:46:05.839478 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:05.839445    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:46:17.722581 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:46:17.722541    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:46:21.121287 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:21.121251    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:46:25.299339 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299288    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:46:25.299735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299701    4515 cpu_manager.go:414] "containerMap: removing container" podUID="6e46f254-6527-40f3-bc5c-3fea46ffef25" containerName="main"
Nov 02 08:46:25.299735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299712    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e46f254-6527-40f3-bc5c-3fea46ffef25" containerName="main"
Nov 02 08:46:25.299735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299723    4515 cpu_manager.go:414] "containerMap: removing container" podUID="d6c0a158-1bce-4737-a8bc-42db65560cbb" containerName="main"
Nov 02 08:46:25.299735 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299728    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c0a158-1bce-4737-a8bc-42db65560cbb" containerName="main"
Nov 02 08:46:25.299897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299779    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e46f254-6527-40f3-bc5c-3fea46ffef25" containerName="main"
Nov 02 08:46:25.299897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.299791    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c0a158-1bce-4737-a8bc-42db65560cbb" containerName="main"
Nov 02 08:46:25.303962 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.303942    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.324867 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.324834    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:46:25.457753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.457703    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t2w\" (UniqueName: \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.457953 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.457830    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.558750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.558657    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t2w\" (UniqueName: \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.558750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.558707    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.559506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.559479    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.583659 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.583628    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t2w\" (UniqueName: \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\") pod \"payload-pre-processing-676c644999-plhvd\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") " pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.616021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.615994    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:25.916685 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:25.916656    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:46:25.916833 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:46:25.916652    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8eabb6_bf7b_48d1_b668_c9b12121bd62.slice/crio-f561d3cdb7e14459bab5b27ff8dbafb07df9699c8a03a760eb81eb5fccdefadd WatchSource:0}: Error finding container f561d3cdb7e14459bab5b27ff8dbafb07df9699c8a03a760eb81eb5fccdefadd: Status 404 returned error can't find the container with id f561d3cdb7e14459bab5b27ff8dbafb07df9699c8a03a760eb81eb5fccdefadd
Nov 02 08:46:26.085510 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:26.085469    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" event={"ID":"dc8eabb6-bf7b-48d1-b668-c9b12121bd62","Type":"ContainerStarted","Data":"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"}
Nov 02 08:46:26.085700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:26.085529    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" event={"ID":"dc8eabb6-bf7b-48d1-b668-c9b12121bd62","Type":"ContainerStarted","Data":"f561d3cdb7e14459bab5b27ff8dbafb07df9699c8a03a760eb81eb5fccdefadd"}
Nov 02 08:46:26.085700 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:26.085567    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:26.128125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:26.128076    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" podStartSLOduration=1.128063888 podStartE2EDuration="1.128063888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:26.12690422 +0000 UTC m=+4930.889341125" watchObservedRunningTime="2026-11-02 08:46:26.128063888 +0000 UTC m=+4930.890500794"
Nov 02 08:46:37.092587 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:37.092557    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:46:37.160065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:37.160029    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 08:46:38.799881 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.799847    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:46:38.803403 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.803373    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:38.825671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.825639    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:46:38.880177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.880135    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:38.880414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.880292    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hx6\" (UniqueName: \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:38.981436 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.981398    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hx6\" (UniqueName: \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:38.981649 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.981492    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:38.982229 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:38.982197    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:39.005866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:39.005839    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hx6\" (UniqueName: \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\") pod \"payload-processing-6bd774d7c8-qp8jq\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") " pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:39.116030 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:39.115937    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:39.416234 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:46:39.416206    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dec009f_740c_438d_8ca3_0ca0d461dd5e.slice/crio-36ac6f95f7253e12868c3ce39c799e572ef591fa556533c75bda53f2797eab5b WatchSource:0}: Error finding container 36ac6f95f7253e12868c3ce39c799e572ef591fa556533c75bda53f2797eab5b: Status 404 returned error can't find the container with id 36ac6f95f7253e12868c3ce39c799e572ef591fa556533c75bda53f2797eab5b
Nov 02 08:46:39.416380 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:39.416262    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:46:40.144671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:40.144635    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" event={"ID":"7dec009f-740c-438d-8ca3-0ca0d461dd5e","Type":"ContainerStarted","Data":"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"}
Nov 02 08:46:40.145217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:40.144688    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" event={"ID":"7dec009f-740c-438d-8ca3-0ca0d461dd5e","Type":"ContainerStarted","Data":"36ac6f95f7253e12868c3ce39c799e572ef591fa556533c75bda53f2797eab5b"}
Nov 02 08:46:40.145217 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:40.144775    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:40.186635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:40.186579    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" podStartSLOduration=2.186565468 podStartE2EDuration="2.186565468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:40.185596341 +0000 UTC m=+4944.948033238" watchObservedRunningTime="2026-11-02 08:46:40.186565468 +0000 UTC m=+4944.949002373"
Nov 02 08:46:42.161001 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.160950    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" containerName="payload-pre-processing" containerID="cri-o://ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320" gracePeriod=25
Nov 02 08:46:42.414200 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.414132    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 08:46:42.513601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.513565    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\") pod \"5622d69a-45ba-47b1-a91c-ca030d136441\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") "
Nov 02 08:46:42.513796 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.513737    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\" (UniqueName: \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\") pod \"5622d69a-45ba-47b1-a91c-ca030d136441\" (UID: \"5622d69a-45ba-47b1-a91c-ca030d136441\") "
Nov 02 08:46:42.513977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.513953    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume" pod "5622d69a-45ba-47b1-a91c-ca030d136441" (UID: "5622d69a-45ba-47b1-a91c-ca030d136441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:46:42.514088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.514074    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5622d69a-45ba-47b1-a91c-ca030d136441-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:42.515825 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.515798    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6" pod "5622d69a-45ba-47b1-a91c-ca030d136441" (UID: "5622d69a-45ba-47b1-a91c-ca030d136441"). InnerVolumeSpecName "kube-api-access-7wcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:46:42.614913 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:42.614879    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wcq6\" (UniqueName: \"kubernetes.io/projected/5622d69a-45ba-47b1-a91c-ca030d136441-kube-api-access-7wcq6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:43.159711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.159675    4515 generic.go:356] "Generic (PLEG): container finished" podID="5622d69a-45ba-47b1-a91c-ca030d136441" containerID="ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320" exitCode=0
Nov 02 08:46:43.159925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.159722    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"
Nov 02 08:46:43.159925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.159766    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" event={"ID":"5622d69a-45ba-47b1-a91c-ca030d136441","Type":"ContainerDied","Data":"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"}
Nov 02 08:46:43.159925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.159819    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" event={"ID":"5622d69a-45ba-47b1-a91c-ca030d136441","Type":"ContainerDied","Data":"a72829309996c53dcde299812b099b11467716510d74d36f3747911f8d5a3bcb"}
Nov 02 08:46:43.159925 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.159841    4515 scope.go:122] "RemoveContainer" containerID="ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"
Nov 02 08:46:43.168903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.168675    4515 scope.go:122] "RemoveContainer" containerID="ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"
Nov 02 08:46:43.169137 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:46:43.168944    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320\": container with ID starting with ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320 not found: ID does not exist" containerID="ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"
Nov 02 08:46:43.169137 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.168969    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320"} err="failed to get container status \"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320\": rpc error: code = NotFound desc = could not find container \"ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320\": container with ID starting with ad558496444c9279aa937c7110f02323dcb276441f401ca687a4aa4b76794320 not found: ID does not exist"
Nov 02 08:46:43.212010 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.211979    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 08:46:43.216155 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.216131    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc"]
Nov 02 08:46:43.366266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.366218    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6dzmc" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.32:50052\" within 1s: context deadline exceeded"
Nov 02 08:46:43.366477 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:46:43.366278    4515 logging.go:55] [core] [Channel #23069 SubChannel #23071]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.32:50052", ServerName: "10.133.0.32:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.32:50052: operation was canceled"
Nov 02 08:46:43.372121 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:43.372096    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" path="/var/lib/kubelet/pods/5622d69a-45ba-47b1-a91c-ca030d136441/volumes"
Nov 02 08:46:47.722279 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:46:47.722236    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:46:51.152912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:51.152883    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:46:51.226087 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:51.226055    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 08:46:53.095300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.095261    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6d9486d65f-f62bl"]
Nov 02 08:46:53.095733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.095680    4515 cpu_manager.go:414] "containerMap: removing container" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" containerName="payload-pre-processing"
Nov 02 08:46:53.095733 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.095693    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" containerName="payload-pre-processing"
Nov 02 08:46:53.095822 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.095751    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5622d69a-45ba-47b1-a91c-ca030d136441" containerName="payload-pre-processing"
Nov 02 08:46:53.100360 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.100339    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.117858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.117834    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6d9486d65f-f62bl"]
Nov 02 08:46:53.219417 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.219381    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7m2s\" (UniqueName: \"kubernetes.io/projected/0da19d73-3cbd-4001-9425-b502bfb43524-kube-api-access-f7m2s\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.219603 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.219463    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da19d73-3cbd-4001-9425-b502bfb43524-plugins-config-volume\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.258527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.258492    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"]
Nov 02 08:46:53.262229 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.262211    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.283907 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.283745    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"]
Nov 02 08:46:53.320227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.320190    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c260812-e09d-47bd-83b0-1dd3b85b5058-plugins-config-volume\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.320470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.320244    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da19d73-3cbd-4001-9425-b502bfb43524-plugins-config-volume\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.320470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.320345    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42bk\" (UniqueName: \"kubernetes.io/projected/9c260812-e09d-47bd-83b0-1dd3b85b5058-kube-api-access-c42bk\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.320470 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.320456    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7m2s\" (UniqueName: \"kubernetes.io/projected/0da19d73-3cbd-4001-9425-b502bfb43524-kube-api-access-f7m2s\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.321059 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.321034    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da19d73-3cbd-4001-9425-b502bfb43524-plugins-config-volume\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.351453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.351385    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7m2s\" (UniqueName: \"kubernetes.io/projected/0da19d73-3cbd-4001-9425-b502bfb43524-kube-api-access-f7m2s\") pod \"payload-processing-6d9486d65f-f62bl\" (UID: \"0da19d73-3cbd-4001-9425-b502bfb43524\") " pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.412626 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.412589    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:53.420928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.420900    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c260812-e09d-47bd-83b0-1dd3b85b5058-plugins-config-volume\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.421056 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.420948    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42bk\" (UniqueName: \"kubernetes.io/projected/9c260812-e09d-47bd-83b0-1dd3b85b5058-kube-api-access-c42bk\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.421595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.421573    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c260812-e09d-47bd-83b0-1dd3b85b5058-plugins-config-volume\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.444763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.444734    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42bk\" (UniqueName: \"kubernetes.io/projected/9c260812-e09d-47bd-83b0-1dd3b85b5058-kube-api-access-c42bk\") pod \"payload-pre-processing-6fcc67b9b4-qld9t\" (UID: \"9c260812-e09d-47bd-83b0-1dd3b85b5058\") " pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.574729 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.574685    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:53.709705 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.709680    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6d9486d65f-f62bl"]
Nov 02 08:46:53.710154 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:46:53.710124    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da19d73_3cbd_4001_9425_b502bfb43524.slice/crio-137f159f5fc1dfb4bb55b72f3780643ddce47a44514f864d1eac52c4e7f87dea WatchSource:0}: Error finding container 137f159f5fc1dfb4bb55b72f3780643ddce47a44514f864d1eac52c4e7f87dea: Status 404 returned error can't find the container with id 137f159f5fc1dfb4bb55b72f3780643ddce47a44514f864d1eac52c4e7f87dea
Nov 02 08:46:53.874513 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:53.874442    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"]
Nov 02 08:46:53.874663 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:46:53.874547    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c260812_e09d_47bd_83b0_1dd3b85b5058.slice/crio-89eb62ab2615a37700cf1f2b033a3544e642e8ff2ea96b6eaa0265b1d9f7a5ea WatchSource:0}: Error finding container 89eb62ab2615a37700cf1f2b033a3544e642e8ff2ea96b6eaa0265b1d9f7a5ea: Status 404 returned error can't find the container with id 89eb62ab2615a37700cf1f2b033a3544e642e8ff2ea96b6eaa0265b1d9f7a5ea
Nov 02 08:46:54.212446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.212345    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t" event={"ID":"9c260812-e09d-47bd-83b0-1dd3b85b5058","Type":"ContainerStarted","Data":"2d7d01704cfdea50c392dbb0210da2d8f243015c03afe2646132403774fcbe16"}
Nov 02 08:46:54.212446 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.212418    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t" event={"ID":"9c260812-e09d-47bd-83b0-1dd3b85b5058","Type":"ContainerStarted","Data":"89eb62ab2615a37700cf1f2b033a3544e642e8ff2ea96b6eaa0265b1d9f7a5ea"}
Nov 02 08:46:54.212951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.212544    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:46:54.213851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.213828    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl" event={"ID":"0da19d73-3cbd-4001-9425-b502bfb43524","Type":"ContainerStarted","Data":"0c43872a5a921550bdacb0680008aed889ead7002704913123c298a989ec298b"}
Nov 02 08:46:54.213938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.213877    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl" event={"ID":"0da19d73-3cbd-4001-9425-b502bfb43524","Type":"ContainerStarted","Data":"137f159f5fc1dfb4bb55b72f3780643ddce47a44514f864d1eac52c4e7f87dea"}
Nov 02 08:46:54.213938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.213920    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:46:54.254487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.254439    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t" podStartSLOduration=1.254426259 podStartE2EDuration="1.254426259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:54.253255061 +0000 UTC m=+4959.015691966" watchObservedRunningTime="2026-11-02 08:46:54.254426259 +0000 UTC m=+4959.016863165"
Nov 02 08:46:54.294670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:54.294618    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl" podStartSLOduration=1.29460583 podStartE2EDuration="1.29460583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:54.294042825 +0000 UTC m=+4959.056479730" watchObservedRunningTime="2026-11-02 08:46:54.29460583 +0000 UTC m=+4959.057042730"
Nov 02 08:46:56.227248 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.227206    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" podUID="dad9dc47-790e-4c20-952e-c2984691c5d4" containerName="payload-processing" containerID="cri-o://8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6" gracePeriod=25
Nov 02 08:46:56.485089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.485028    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 08:46:56.554463 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.554424    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\" (UniqueName: \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\") pod \"dad9dc47-790e-4c20-952e-c2984691c5d4\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") "
Nov 02 08:46:56.554691 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.554477    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\") pod \"dad9dc47-790e-4c20-952e-c2984691c5d4\" (UID: \"dad9dc47-790e-4c20-952e-c2984691c5d4\") "
Nov 02 08:46:56.554919 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.554888    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume" pod "dad9dc47-790e-4c20-952e-c2984691c5d4" (UID: "dad9dc47-790e-4c20-952e-c2984691c5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:46:56.556701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.556675    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg" pod "dad9dc47-790e-4c20-952e-c2984691c5d4" (UID: "dad9dc47-790e-4c20-952e-c2984691c5d4"). InnerVolumeSpecName "kube-api-access-52sqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:46:56.655504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.655464    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52sqg\" (UniqueName: \"kubernetes.io/projected/dad9dc47-790e-4c20-952e-c2984691c5d4-kube-api-access-52sqg\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:56.655504 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:56.655502    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad9dc47-790e-4c20-952e-c2984691c5d4-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:46:57.229115 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.229069    4515 generic.go:356] "Generic (PLEG): container finished" podID="dad9dc47-790e-4c20-952e-c2984691c5d4" containerID="8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6" exitCode=0
Nov 02 08:46:57.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.229120    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"
Nov 02 08:46:57.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.229128    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" event={"ID":"dad9dc47-790e-4c20-952e-c2984691c5d4","Type":"ContainerDied","Data":"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"}
Nov 02 08:46:57.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.229204    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwdh9" event={"ID":"dad9dc47-790e-4c20-952e-c2984691c5d4","Type":"ContainerDied","Data":"5128135d0f857be6c752e566db8a1fdc97747778475b98651c6ba4a214e29506"}
Nov 02 08:46:57.229576 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.229229    4515 scope.go:122] "RemoveContainer" containerID="8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"
Nov 02 08:46:57.237750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.237728    4515 scope.go:122] "RemoveContainer" containerID="8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"
Nov 02 08:46:57.238017 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:46:57.237997    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6\": container with ID starting with 8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6 not found: ID does not exist" containerID="8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"
Nov 02 08:46:57.238136 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.238023    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6"} err="failed to get container status \"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6\": rpc error: code = NotFound desc = could not find container \"8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6\": container with ID starting with 8b0bbd08f47442878b3038015973589c9b4fa34772e50b09648a5aaa1d4446c6 not found: ID does not exist"
Nov 02 08:46:57.280792 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.280756    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 08:46:57.293542 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.293511    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwdh9"]
Nov 02 08:46:57.372577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:46:57.372544    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad9dc47-790e-4c20-952e-c2984691c5d4" path="/var/lib/kubelet/pods/dad9dc47-790e-4c20-952e-c2984691c5d4/volumes"
Nov 02 08:47:05.222040 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:05.222004    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6d9486d65f-f62bl"
Nov 02 08:47:05.222466 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:05.222081    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6fcc67b9b4-qld9t"
Nov 02 08:47:05.303232 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:05.303195    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:47:05.342746 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:05.342707    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:47:08.794414 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.794374    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:47:08.794976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.794848    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dad9dc47-790e-4c20-952e-c2984691c5d4" containerName="payload-processing"
Nov 02 08:47:08.794976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.794864    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad9dc47-790e-4c20-952e-c2984691c5d4" containerName="payload-processing"
Nov 02 08:47:08.794976 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.794936    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad9dc47-790e-4c20-952e-c2984691c5d4" containerName="payload-processing"
Nov 02 08:47:08.797585 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.797569    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:08.813224 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.813197    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 02 08:47:08.825797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.825754    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 08:47:08.825797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.825761    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 08:47:08.825797 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.825789    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 08:47:08.826054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.825857    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nwfhf\""
Nov 02 08:47:08.826054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.825913    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 08:47:08.827679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.827660    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:47:08.980197 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.980151    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:08.980428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.980237    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5mv\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:08.980428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.980337    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:08.980428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:08.980380    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.081241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.081146    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.081241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.081198    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5mv\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.081241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.081215    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.081241 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.081232    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.081961 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.081936    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.082095 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.082073    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.083840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.083811    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.106611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.106580    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5mv\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\") pod \"payload-processing-external-model-5bcd7997-78tsw\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.110396 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.110378    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:09.419843 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:47:09.419807    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2f20e2_856e_43e0_8ba1_dfe4613b2066.slice/crio-de47a039ed71f75b44fdb741b655a349c07b9a76d26f07027ced7460e3556f45 WatchSource:0}: Error finding container de47a039ed71f75b44fdb741b655a349c07b9a76d26f07027ced7460e3556f45: Status 404 returned error can't find the container with id de47a039ed71f75b44fdb741b655a349c07b9a76d26f07027ced7460e3556f45
Nov 02 08:47:09.419843 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.419835    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:47:09.982563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:09.982530    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:47:10.287628 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.287586    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" event={"ID":"ae2f20e2-856e-43e0-8ba1-dfe4613b2066","Type":"ContainerStarted","Data":"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"}
Nov 02 08:47:10.287791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.287648    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" event={"ID":"ae2f20e2-856e-43e0-8ba1-dfe4613b2066","Type":"ContainerStarted","Data":"de47a039ed71f75b44fdb741b655a349c07b9a76d26f07027ced7460e3556f45"}
Nov 02 08:47:10.287791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.287701    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:10.304505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.304470    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" podUID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" containerName="payload-processing" containerID="cri-o://a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030" gracePeriod=25
Nov 02 08:47:10.330900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.330858    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" podStartSLOduration=2.330845797 podStartE2EDuration="2.330845797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:47:10.330211616 +0000 UTC m=+4975.092648522" watchObservedRunningTime="2026-11-02 08:47:10.330845797 +0000 UTC m=+4975.093282703"
Nov 02 08:47:10.343471 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.343420    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" podUID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" containerName="payload-pre-processing" containerID="cri-o://bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da" gracePeriod=25
Nov 02 08:47:10.635058 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.635022    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:47:10.696617 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.696572    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\" (UniqueName: \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\") pod \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") "
Nov 02 08:47:10.696855 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.696718    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\") pod \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\" (UID: \"7dec009f-740c-438d-8ca3-0ca0d461dd5e\") "
Nov 02 08:47:10.697551 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.697522    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume" pod "7dec009f-740c-438d-8ca3-0ca0d461dd5e" (UID: "7dec009f-740c-438d-8ca3-0ca0d461dd5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:47:10.698373 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.698346    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:47:10.700180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.700146    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6" pod "7dec009f-740c-438d-8ca3-0ca0d461dd5e" (UID: "7dec009f-740c-438d-8ca3-0ca0d461dd5e"). InnerVolumeSpecName "kube-api-access-88hx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:47:10.797671 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.797627    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\" (UniqueName: \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\") pod \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") "
Nov 02 08:47:10.797886 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.797687    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\") pod \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\" (UID: \"dc8eabb6-bf7b-48d1-b668-c9b12121bd62\") "
Nov 02 08:47:10.797991 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.797973    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dec009f-740c-438d-8ca3-0ca0d461dd5e-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:47:10.798084 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.797997    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hx6\" (UniqueName: \"kubernetes.io/projected/7dec009f-740c-438d-8ca3-0ca0d461dd5e-kube-api-access-88hx6\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:47:10.798450 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.798423    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume" pod "dc8eabb6-bf7b-48d1-b668-c9b12121bd62" (UID: "dc8eabb6-bf7b-48d1-b668-c9b12121bd62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:47:10.800577 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.800549    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w" pod "dc8eabb6-bf7b-48d1-b668-c9b12121bd62" (UID: "dc8eabb6-bf7b-48d1-b668-c9b12121bd62"). InnerVolumeSpecName "kube-api-access-z8t2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:47:10.898944 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.898837    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8t2w\" (UniqueName: \"kubernetes.io/projected/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-kube-api-access-z8t2w\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:47:10.899134 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:10.898886    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8eabb6-bf7b-48d1-b668-c9b12121bd62-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:47:11.296153 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.296109    4515 generic.go:356] "Generic (PLEG): container finished" podID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" containerID="bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da" exitCode=0
Nov 02 08:47:11.296708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.296169    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd"
Nov 02 08:47:11.296708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.296199    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" event={"ID":"dc8eabb6-bf7b-48d1-b668-c9b12121bd62","Type":"ContainerDied","Data":"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"}
Nov 02 08:47:11.296708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.296268    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-676c644999-plhvd" event={"ID":"dc8eabb6-bf7b-48d1-b668-c9b12121bd62","Type":"ContainerDied","Data":"f561d3cdb7e14459bab5b27ff8dbafb07df9699c8a03a760eb81eb5fccdefadd"}
Nov 02 08:47:11.296708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.296302    4515 scope.go:122] "RemoveContainer" containerID="bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"
Nov 02 08:47:11.298616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.298435    4515 generic.go:356] "Generic (PLEG): container finished" podID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" containerID="a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030" exitCode=0
Nov 02 08:47:11.298616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.298484    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq"
Nov 02 08:47:11.298616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.298533    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" event={"ID":"7dec009f-740c-438d-8ca3-0ca0d461dd5e","Type":"ContainerDied","Data":"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"}
Nov 02 08:47:11.298616 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.298577    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bd774d7c8-qp8jq" event={"ID":"7dec009f-740c-438d-8ca3-0ca0d461dd5e","Type":"ContainerDied","Data":"36ac6f95f7253e12868c3ce39c799e572ef591fa556533c75bda53f2797eab5b"}
Nov 02 08:47:11.314726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.314612    4515 scope.go:122] "RemoveContainer" containerID="bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"
Nov 02 08:47:11.315012 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:47:11.314965    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da\": container with ID starting with bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da not found: ID does not exist" containerID="bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"
Nov 02 08:47:11.315112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.315004    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da"} err="failed to get container status \"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da\": rpc error: code = NotFound desc = could not find container \"bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da\": container with ID starting with bfc02e74c154de478edba5131a667a7d4f6935c85b4f9932ebedfb331f2234da not found: ID does not exist"
Nov 02 08:47:11.315112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.315026    4515 scope.go:122] "RemoveContainer" containerID="a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"
Nov 02 08:47:11.327749 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.327712    4515 scope.go:122] "RemoveContainer" containerID="a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"
Nov 02 08:47:11.328216 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:47:11.328123    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030\": container with ID starting with a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030 not found: ID does not exist" containerID="a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"
Nov 02 08:47:11.328216 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.328164    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030"} err="failed to get container status \"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030\": rpc error: code = NotFound desc = could not find container \"a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030\": container with ID starting with a675c8aa2d2dc071e1fdaca65b06a1396525eed6368f37fd41b4d9926ddfc030 not found: ID does not exist"
Nov 02 08:47:11.376302 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.376229    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:47:11.385814 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.385776    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-6bd774d7c8-qp8jq"]
Nov 02 08:47:11.420227 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.420172    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:47:11.426338 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:11.426288    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-676c644999-plhvd"]
Nov 02 08:47:12.552333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.552066    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:47:12.552333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.552145    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:47:12.605333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.599429    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:47:12.605333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.599520    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:47:12.774334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.769437    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:47:12.774334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:12.769555    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:47:13.390331 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:13.389991    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" path="/var/lib/kubelet/pods/7dec009f-740c-438d-8ca3-0ca0d461dd5e/volumes"
Nov 02 08:47:13.393268 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:13.390726    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" path="/var/lib/kubelet/pods/dc8eabb6-bf7b-48d1-b668-c9b12121bd62/volumes"
Nov 02 08:47:17.722797 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:47:17.722760    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:47:21.303097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:47:21.303066    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:47:47.722551 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:47:47.722471    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:48:17.722139 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:48:17.722092    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:48:47.721543 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:48:47.721501    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:48:48.091529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:48.091490    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:48:49.596411 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:49.596377    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:48:53.091882 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.091802    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" podUID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" containerName="payload-processing" containerID="cri-o://8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408" gracePeriod=25
Nov 02 08:48:53.338028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.338001    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:48:53.428355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428251    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\") pod \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") "
Nov 02 08:48:53.428355 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428292    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\") pod \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") "
Nov 02 08:48:53.428594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428403    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\") pod \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") "
Nov 02 08:48:53.428594 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428470    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\") pod \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\" (UID: \"ae2f20e2-856e-43e0-8ba1-dfe4613b2066\") "
Nov 02 08:48:53.428727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428697    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume" pod "ae2f20e2-856e-43e0-8ba1-dfe4613b2066" (UID: "ae2f20e2-856e-43e0-8ba1-dfe4613b2066"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:48:53.428727 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428704    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume" pod "ae2f20e2-856e-43e0-8ba1-dfe4613b2066" (UID: "ae2f20e2-856e-43e0-8ba1-dfe4613b2066"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:48:53.428846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428805    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:48:53.428846 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.428827    4515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-routing-overlay-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:48:53.430601 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.430578    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials" pod "ae2f20e2-856e-43e0-8ba1-dfe4613b2066" (UID: "ae2f20e2-856e-43e0-8ba1-dfe4613b2066"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:48:53.430678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.430614    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv" pod "ae2f20e2-856e-43e0-8ba1-dfe4613b2066" (UID: "ae2f20e2-856e-43e0-8ba1-dfe4613b2066"). InnerVolumeSpecName "kube-api-access-wh5mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:48:53.530708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.530668    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5mv\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-kube-api-access-wh5mv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:48:53.530708 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.530700    4515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae2f20e2-856e-43e0-8ba1-dfe4613b2066-provider-credentials\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:48:53.806921 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.806888    4515 generic.go:356] "Generic (PLEG): container finished" podID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" containerID="8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408" exitCode=0
Nov 02 08:48:53.807159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.806935    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"
Nov 02 08:48:53.807159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.806972    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" event={"ID":"ae2f20e2-856e-43e0-8ba1-dfe4613b2066","Type":"ContainerDied","Data":"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"}
Nov 02 08:48:53.807159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.807033    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-78tsw" event={"ID":"ae2f20e2-856e-43e0-8ba1-dfe4613b2066","Type":"ContainerDied","Data":"de47a039ed71f75b44fdb741b655a349c07b9a76d26f07027ced7460e3556f45"}
Nov 02 08:48:53.807159 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.807047    4515 scope.go:122] "RemoveContainer" containerID="8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"
Nov 02 08:48:53.817488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.817469    4515 scope.go:122] "RemoveContainer" containerID="8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"
Nov 02 08:48:53.817803 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:48:53.817777    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408\": container with ID starting with 8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408 not found: ID does not exist" containerID="8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"
Nov 02 08:48:53.817938 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.817806    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408"} err="failed to get container status \"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408\": rpc error: code = NotFound desc = could not find container \"8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408\": container with ID starting with 8cfaf250fff83ebc0a5fdc75e6ec3e001b2deb8393fb43c6ab912f58c0912408 not found: ID does not exist"
Nov 02 08:48:53.828466 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:48:53.828444    4515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2f20e2_856e_43e0_8ba1_dfe4613b2066.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:48:53.881016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.880984    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:48:53.888675 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:53.888644    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-78tsw"]
Nov 02 08:48:55.371946 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:55.371917    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" path="/var/lib/kubelet/pods/ae2f20e2-856e-43e0-8ba1-dfe4613b2066/volumes"
Nov 02 08:48:56.347215 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:48:56.347174    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:49:12.106954 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:49:12.106910    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:49:16.807135 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:49:16.807096    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:49:17.736389 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:49:17.736302    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:49:32.049180 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:49:32.049138    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:49:47.721808 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:49:47.721767    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:49:47.877764 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:49:47.877733    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:50:10.940453 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:50:10.940416    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:50:17.722906 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:50:17.722861    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:50:21.250088 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:50:21.250049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:50:25.945968 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:50:25.945930    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:50:42.843234 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:50:42.843146    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:50:47.794914 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:50:47.794867    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:50:48.183139 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:50:48.183044    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:09.924794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:09.924757    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:14.539185 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:14.539147    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:17.778022 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:51:17.777977    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:51:20.621752 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:20.621716    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:27.643191 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:27.643151    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:33.135318 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:33.135282    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:47.722370 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:51:47.722327    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:51:51.423196 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:51.423160    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:57.506449 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:57.506407    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:51:57.507213 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:57.507178    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:51:57.541330 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:57.534395    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:51:57.592506 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:57.592398    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:51:57.592784 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:51:57.592747    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:52:10.516278 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:10.516242    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:52:16.041160 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:16.041121    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:52:17.722782 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:52:17.722743    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:52:22.332984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:22.332948    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:52:40.202595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:40.202560    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:52:42.510487 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.510437    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:52:42.511221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.511180    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:52:42.553502 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.553452    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:52:42.553870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.553830    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:52:42.598559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.598404    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:52:42.598559 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.598488    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:52:42.764539 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.764434    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:52:42.764865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:42.764832    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:52:47.721403 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:52:47.721364    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:52:50.540208 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:52:50.540171    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:53:07.176301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:07.176264    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:53:10.642480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642442    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642847    4515 cpu_manager.go:414] "containerMap: removing container" podUID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" containerName="payload-processing"
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642858    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" containerName="payload-processing"
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642879    4515 cpu_manager.go:414] "containerMap: removing container" podUID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" containerName="payload-processing"
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642885    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" containerName="payload-processing"
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642898    4515 cpu_manager.go:414] "containerMap: removing container" podUID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" containerName="payload-pre-processing"
Nov 02 08:53:10.642908 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642905    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" containerName="payload-pre-processing"
Nov 02 08:53:10.643299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642978    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8eabb6-bf7b-48d1-b668-c9b12121bd62" containerName="payload-pre-processing"
Nov 02 08:53:10.643299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642990    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2f20e2-856e-43e0-8ba1-dfe4613b2066" containerName="payload-processing"
Nov 02 08:53:10.643299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.642998    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dec009f-740c-438d-8ca3-0ca0d461dd5e" containerName="payload-processing"
Nov 02 08:53:10.646611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.646592    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.661893 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.661863    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-5c7145-openshift-default-dockercfg-mrs4n\""
Nov 02 08:53:10.676496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.676466    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:53:10.724558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724522    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724575    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724599    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724698    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5m9d\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724731    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724756    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724775    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724898    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.724967 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.724932    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.825737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825695    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.825737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825741    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825787    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5m9d\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825816    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825854    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825880    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825952    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826026 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.825988    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826408 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826046    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826408 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826280    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826527 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826502    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826523    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826592 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826572    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.826873 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.826852    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.828214 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.828190    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.828753 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.828735    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.849188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.849161    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.852454 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.852427    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5m9d\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\") pod \"e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") " pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:10.962703 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:10.962610    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:11.278728 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:53:11.278694    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacf1143_0ba7_4f10_a8e2_82bf7b719c48.slice/crio-3bd1fe12ca0616825ef4c1051ed74f3fc40efaf45ee7d4d3552be5ada2952baf WatchSource:0}: Error finding container 3bd1fe12ca0616825ef4c1051ed74f3fc40efaf45ee7d4d3552be5ada2952baf: Status 404 returned error can't find the container with id 3bd1fe12ca0616825ef4c1051ed74f3fc40efaf45ee7d4d3552be5ada2952baf
Nov 02 08:53:11.278848 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:11.278820    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:53:11.280813 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:11.280776    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:11.280912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:11.280843    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:11.280912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:11.280871    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:11.327660 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:11.327631    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerStarted","Data":"3bd1fe12ca0616825ef4c1051ed74f3fc40efaf45ee7d4d3552be5ada2952baf"}
Nov 02 08:53:12.333300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:12.333257    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerStarted","Data":"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"}
Nov 02 08:53:12.393686 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:12.393630    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podStartSLOduration=2.393616638 podStartE2EDuration="2.393616638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:53:12.391645953 +0000 UTC m=+5337.154082859" watchObservedRunningTime="2026-11-02 08:53:12.393616638 +0000 UTC m=+5337.156053544"
Nov 02 08:53:12.963068 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:12.963023    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:12.968490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:12.968462    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:13.338264 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:13.338230    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:13.339377 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:13.339352    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:13.926734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:13.926698    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:53:16.359262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.359230    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:53:16.362833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.362804    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.379381 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.379350    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.379538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.379389    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmqx\" (UniqueName: \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.379538 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.379408    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.379711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.379678    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.400451 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.400420    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5c7145-serving-cert\""
Nov 02 08:53:16.405690 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.405656    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:53:16.481012 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.480968    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.481233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.481020    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmqx\" (UniqueName: \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.481233 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.481051    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.481498 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.481471    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.481720 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:53:16.481170    4515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-serving-cert: secret "maas-api-e2e-worker-single-main-5c7145-serving-cert" not found
Nov 02 08:53:16.481805 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:53:16.481778    4515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls podName:c268428a-2bc8-4b88-94c0-032e828871e1 nodeName:}" failed. No retries permitted until 2026-11-02 08:53:16.981753483 +0000 UTC m=+5341.744190368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls") pod "maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" (UID: "c268428a-2bc8-4b88-94c0-032e828871e1") : secret "maas-api-e2e-worker-single-main-5c7145-serving-cert" not found
Nov 02 08:53:16.481876 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.481828    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.484488 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.484461    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.505833 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.505800    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmqx\" (UniqueName: \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.730865 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.730783    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:53:16.734491 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.734474    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.753977 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.753952    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-89e08e1e-dockercfg-mt57r\""
Nov 02 08:53:16.754750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.754729    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5c7145\""
Nov 02 08:53:16.757523 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.757498    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:53:16.785635 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.785578    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzkv\" (UniqueName: \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.785826 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.785666    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.822756 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.820968    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:53:16.825851 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.825826    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.848948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.848918    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:53:16.886620 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.886582    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.886819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.886626    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh4c\" (UniqueName: \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.886819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.886702    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.886951 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.886816    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzkv\" (UniqueName: \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.887280 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.887262    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.910606 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.910573    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzkv\" (UniqueName: \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:16.987458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.987365    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.987458 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.987456    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh4c\" (UniqueName: \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.987713 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.987496    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:16.988097 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.988074    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:16.989840 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:16.989816    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:17.016726 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.016701    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh4c\" (UniqueName: \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:17.047737 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.047701    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:17.139745 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.139671    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:17.277090 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.277050    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:17.346252 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.346222    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:53:17.346943 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:53:17.346912    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57e794b_3e9f_4750_a277_3caf231be071.slice/crio-a4b9bdac73d27213b21cea9457f4afa390ea3e8f4281dcef20a61ca975b2980d WatchSource:0}: Error finding container a4b9bdac73d27213b21cea9457f4afa390ea3e8f4281dcef20a61ca975b2980d: Status 404 returned error can't find the container with id a4b9bdac73d27213b21cea9457f4afa390ea3e8f4281dcef20a61ca975b2980d
Nov 02 08:53:17.358857 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.358823    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" event={"ID":"a57e794b-3e9f-4750-a277-3caf231be071","Type":"ContainerStarted","Data":"a4b9bdac73d27213b21cea9457f4afa390ea3e8f4281dcef20a61ca975b2980d"}
Nov 02 08:53:17.437447 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:53:17.437417    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c8c7fd_d512_47ee_850b_858e6402dd87.slice/crio-d20e5c5fe7ea79171233bd2562a7b81326e44ff5235caf517b075813233c751d WatchSource:0}: Error finding container d20e5c5fe7ea79171233bd2562a7b81326e44ff5235caf517b075813233c751d: Status 404 returned error can't find the container with id d20e5c5fe7ea79171233bd2562a7b81326e44ff5235caf517b075813233c751d
Nov 02 08:53:17.437754 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.437492    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:53:17.581563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.581466    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:53:17.586583 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:53:17.586551    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc268428a_2bc8_4b88_94c0_032e828871e1.slice/crio-efc5f5ea4dba648f1456f98a1fee24ca5982c3477d5cc380f8a2d573b37f2b13 WatchSource:0}: Error finding container efc5f5ea4dba648f1456f98a1fee24ca5982c3477d5cc380f8a2d573b37f2b13: Status 404 returned error can't find the container with id efc5f5ea4dba648f1456f98a1fee24ca5982c3477d5cc380f8a2d573b37f2b13
Nov 02 08:53:17.588147 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:17.588129    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 08:53:17.722493 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:53:17.722449    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:53:18.365061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.365015    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" event={"ID":"a57e794b-3e9f-4750-a277-3caf231be071","Type":"ContainerStarted","Data":"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"}
Nov 02 08:53:18.365254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.365084    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:18.366597 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.366572    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" event={"ID":"a8c8c7fd-d512-47ee-850b-858e6402dd87","Type":"ContainerStarted","Data":"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"}
Nov 02 08:53:18.366760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.366618    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" event={"ID":"a8c8c7fd-d512-47ee-850b-858e6402dd87","Type":"ContainerStarted","Data":"d20e5c5fe7ea79171233bd2562a7b81326e44ff5235caf517b075813233c751d"}
Nov 02 08:53:18.366760 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.366642    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:18.368184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.368160    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" event={"ID":"c268428a-2bc8-4b88-94c0-032e828871e1","Type":"ContainerStarted","Data":"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"}
Nov 02 08:53:18.368290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.368203    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" event={"ID":"c268428a-2bc8-4b88-94c0-032e828871e1","Type":"ContainerStarted","Data":"efc5f5ea4dba648f1456f98a1fee24ca5982c3477d5cc380f8a2d573b37f2b13"}
Nov 02 08:53:18.368290 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.368225    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:18.407654 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.407605    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" podStartSLOduration=2.407591833 podStartE2EDuration="2.407591833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:53:18.406656788 +0000 UTC m=+5343.169093694" watchObservedRunningTime="2026-11-02 08:53:18.407591833 +0000 UTC m=+5343.170028780"
Nov 02 08:53:18.450177 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.450122    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" podStartSLOduration=2.08620591 podStartE2EDuration="2.450108727s" totalImagesPullingTime="363.902817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:53:18.448837386 +0000 UTC m=+5343.211274291" watchObservedRunningTime="2026-11-02 08:53:18.450108727 +0000 UTC m=+5343.212545657"
Nov 02 08:53:18.491942 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:18.491885    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" podStartSLOduration=2.49187105 podStartE2EDuration="2.49187105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:53:18.48994568 +0000 UTC m=+5343.252382585" watchObservedRunningTime="2026-11-02 08:53:18.49187105 +0000 UTC m=+5343.254307956"
Nov 02 08:53:19.918117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:19.918079    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:53:24.377045 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:24.377014    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:53:29.374763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:29.374683    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:53:29.374763 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:29.374762    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:53:31.320897 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.320859    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:53:31.331863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.328408    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.357625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.357584    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:53:31.367684 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.367646    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5c7145\"/\"simulator-single-main-5c7145-kserve-self-signed-certs\""
Nov 02 08:53:31.367877 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.367801    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5c7145\"/\"default-dockercfg-7ptcl\""
Nov 02 08:53:31.423125 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.423087    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5c7145\"/\"openshift-service-ca.crt\""
Nov 02 08:53:31.423357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.423253    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5c7145\"/\"kube-root-ca.crt\""
Nov 02 08:53:31.463879 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.463831    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5ng\" (UniqueName: \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.464093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.463891    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.464093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.464031    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.464093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.464059    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.464258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.464169    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.464258 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.464202    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.566221 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566170    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5ng\" (UniqueName: \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566521    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566630    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566657    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566734    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566840    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.566949    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.567103    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.567265 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.567220    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.570300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.570061    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.570300 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.570264    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.593556 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.593474    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5ng\" (UniqueName: \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\") pod \"simulator-single-main-5c7145-kserve-6648c88b69-fnbhs\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") " pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.672670 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.672630    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:31.987264 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:53:31.987233    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe34b4ed_614d_4171_a6b7_fba49d709389.slice/crio-ad8aeda6d7a3798d49836b650382f03560e664c34d0174a1a6eca205f21ea238 WatchSource:0}: Error finding container ad8aeda6d7a3798d49836b650382f03560e664c34d0174a1a6eca205f21ea238: Status 404 returned error can't find the container with id ad8aeda6d7a3798d49836b650382f03560e664c34d0174a1a6eca205f21ea238
Nov 02 08:53:31.987415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:31.987335    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:53:32.453579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:32.453539    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" event={"ID":"fe34b4ed-614d-4171-a6b7-fba49d709389","Type":"ContainerStarted","Data":"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"}
Nov 02 08:53:32.454069 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:32.453589    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" event={"ID":"fe34b4ed-614d-4171-a6b7-fba49d709389","Type":"ContainerStarted","Data":"ad8aeda6d7a3798d49836b650382f03560e664c34d0174a1a6eca205f21ea238"}
Nov 02 08:53:32.500512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:32.500454    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" podStartSLOduration=1.5004420440000001 podStartE2EDuration="1.500442044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:53:32.499008117 +0000 UTC m=+5357.261445036" watchObservedRunningTime="2026-11-02 08:53:32.500442044 +0000 UTC m=+5357.262878951"
Nov 02 08:53:32.672948 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:32.672911    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:33.748389 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:33.748295    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:53:34.465439 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.465402    4515 generic.go:356] "Generic (PLEG): container finished" podID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerID="2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60" exitCode=0
Nov 02 08:53:34.465695 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.465485    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerDied","Data":"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"}
Nov 02 08:53:34.465972 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.465951    4515 scope.go:122] "RemoveContainer" containerID="2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"
Nov 02 08:53:34.467292 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.467261    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:34.467445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.467376    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:34.467445 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:34.467431    4515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:53:35.471625 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:35.471587    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerStarted","Data":"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"}
Nov 02 08:53:35.963810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:35.963776    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:35.964116 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:35.964089    4515 patch_prober.go:29] interesting pod/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 08:53:35.964181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:35.964159    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 08:53:36.963512 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:36.963475    4515 patch_prober.go:29] interesting pod/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 08:53:36.963924 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:36.963528    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 08:53:37.963871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:37.963825    4515 patch_prober.go:29] interesting pod/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 08:53:37.964319 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:37.963897    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 08:53:38.963080 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:38.963037    4515 patch_prober.go:29] interesting pod/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 08:53:38.963279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:38.963098    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 08:53:39.963364 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:39.963298    4515 patch_prober.go:29] interesting pod/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 08:53:39.963778 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:39.963392    4515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 08:53:40.967270 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:40.967238    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:40.967698 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:40.967551    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:40.968169 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:40.968147    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:53:41.673073 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:41.673033    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:41.685610 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:41.685578    4515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:42.514483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:42.514450    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:53:47.722701 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:53:47.722655    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:53:51.778739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:53:51.778702    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:54:16.852262 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:16.852220    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:54:17.723586 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:17.723539    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:54:20.412295 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:20.412260    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:54:21.242042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:21.242005    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:54:21.311891 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:21.311855    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:54:23.617028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:23.616992    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:54:23.617503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:23.617228    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" podUID="c268428a-2bc8-4b88-94c0-032e828871e1" containerName="maas-api" containerID="cri-o://83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e" gracePeriod=30
Nov 02 08:54:24.088815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.088792    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:54:24.121468 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121433    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\") pod \"c268428a-2bc8-4b88-94c0-032e828871e1\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") "
Nov 02 08:54:24.121637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121529    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\") pod \"c268428a-2bc8-4b88-94c0-032e828871e1\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") "
Nov 02 08:54:24.121637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121575    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\" (UniqueName: \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\") pod \"c268428a-2bc8-4b88-94c0-032e828871e1\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") "
Nov 02 08:54:24.121637 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121617    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") pod \"c268428a-2bc8-4b88-94c0-032e828871e1\" (UID: \"c268428a-2bc8-4b88-94c0-032e828871e1\") "
Nov 02 08:54:24.121928 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121898    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca" pod "c268428a-2bc8-4b88-94c0-032e828871e1" (UID: "c268428a-2bc8-4b88-94c0-032e828871e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:54:24.122065 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.121993    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268428a-2bc8-4b88-94c0-032e828871e1-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:24.123696 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.123671    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs" pod "c268428a-2bc8-4b88-94c0-032e828871e1" (UID: "c268428a-2bc8-4b88-94c0-032e828871e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:54:24.123819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.123737    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls" pod "c268428a-2bc8-4b88-94c0-032e828871e1" (UID: "c268428a-2bc8-4b88-94c0-032e828871e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:54:24.123989 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.123972    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx" pod "c268428a-2bc8-4b88-94c0-032e828871e1" (UID: "c268428a-2bc8-4b88-94c0-032e828871e1"). InnerVolumeSpecName "kube-api-access-jdmqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:54:24.223598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.223512    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:24.223598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.223544    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmqx\" (UniqueName: \"kubernetes.io/projected/c268428a-2bc8-4b88-94c0-032e828871e1-kube-api-access-jdmqx\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:24.223598 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.223555    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268428a-2bc8-4b88-94c0-032e828871e1-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:24.695320 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.695278    4515 generic.go:356] "Generic (PLEG): container finished" podID="c268428a-2bc8-4b88-94c0-032e828871e1" containerID="83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e" exitCode=0
Nov 02 08:54:24.695795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.695359    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"
Nov 02 08:54:24.695795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.695372    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" event={"ID":"c268428a-2bc8-4b88-94c0-032e828871e1","Type":"ContainerDied","Data":"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"}
Nov 02 08:54:24.695795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.695429    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2" event={"ID":"c268428a-2bc8-4b88-94c0-032e828871e1","Type":"ContainerDied","Data":"efc5f5ea4dba648f1456f98a1fee24ca5982c3477d5cc380f8a2d573b37f2b13"}
Nov 02 08:54:24.695795 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.695450    4515 scope.go:122] "RemoveContainer" containerID="83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"
Nov 02 08:54:24.705199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.705179    4515 scope.go:122] "RemoveContainer" containerID="83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"
Nov 02 08:54:24.705493 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:24.705473    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e\": container with ID starting with 83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e not found: ID does not exist" containerID="83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"
Nov 02 08:54:24.705563 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.705508    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e"} err="failed to get container status \"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e\": rpc error: code = NotFound desc = could not find container \"83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e\": container with ID starting with 83efcb2b94a3ecf4835c2f338692a88d9b89a6efd8f2238b2bb7fede9f5abe7e not found: ID does not exist"
Nov 02 08:54:24.747679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.747617    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:54:24.760337 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:24.760299    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-9zlf2"]
Nov 02 08:54:25.373064 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:25.373026    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c268428a-2bc8-4b88-94c0-032e828871e1" path="/var/lib/kubelet/pods/c268428a-2bc8-4b88-94c0-032e828871e1/volumes"
Nov 02 08:54:26.044091 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:26.044048    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:54:26.243243 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:26.243199    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing" containerID="cri-o://54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6" gracePeriod=25
Nov 02 08:54:26.313027 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:26.312924    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing" containerID="cri-o://bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b" gracePeriod=25
Nov 02 08:54:29.373089 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:29.373049    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:54:29.373552 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:29.373125    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:54:35.432614 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.432542    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerName="main" containerID="cri-o://3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981" gracePeriod=45
Nov 02 08:54:35.503188 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.503148    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerName="main" probeResult="failure" output="Get \"https://10.133.1.86:8000/ready\": dial tcp 10.133.1.86:8000: connect: connection refused"
Nov 02 08:54:35.689866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.689807    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:54:35.749061 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.749030    4515 generic.go:356] "Generic (PLEG): container finished" podID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerID="3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981" exitCode=0
Nov 02 08:54:35.749253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.749077    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"
Nov 02 08:54:35.749253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.749093    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" event={"ID":"fe34b4ed-614d-4171-a6b7-fba49d709389","Type":"ContainerDied","Data":"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"}
Nov 02 08:54:35.749253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.749164    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs" event={"ID":"fe34b4ed-614d-4171-a6b7-fba49d709389","Type":"ContainerDied","Data":"ad8aeda6d7a3798d49836b650382f03560e664c34d0174a1a6eca205f21ea238"}
Nov 02 08:54:35.749253 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.749189    4515 scope.go:122] "RemoveContainer" containerID="3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"
Nov 02 08:54:35.758581 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.758561    4515 scope.go:122] "RemoveContainer" containerID="3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"
Nov 02 08:54:35.758931 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:35.758901    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981\": container with ID starting with 3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981 not found: ID does not exist" containerID="3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"
Nov 02 08:54:35.759042 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.758943    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981"} err="failed to get container status \"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981\": rpc error: code = NotFound desc = could not find container \"3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981\": container with ID starting with 3a1efded55559b9b4a3cc2da4d97d0feeb05c357e422ec31e9514c00909a6981 not found: ID does not exist"
Nov 02 08:54:35.838481 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838449    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838518    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838546    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\" (UniqueName: \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838598    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838663 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838644    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838682    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\") pod \"fe34b4ed-614d-4171-a6b7-fba49d709389\" (UID: \"fe34b4ed-614d-4171-a6b7-fba49d709389\") "
Nov 02 08:54:35.838912 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838746    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:54:35.839028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838952    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:54:35.839028 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.838962    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:54:35.839218 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.839198    4515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-tmp-dir\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:35.839281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.839222    4515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-home\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:35.839281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.839236    4515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-model-cache\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:35.840772 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.840749    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:54:35.841062 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.840892    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:54:35.841164 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.841073    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng" pod "fe34b4ed-614d-4171-a6b7-fba49d709389" (UID: "fe34b4ed-614d-4171-a6b7-fba49d709389"). InnerVolumeSpecName "kube-api-access-vk5ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:54:35.940611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.940527    4515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe34b4ed-614d-4171-a6b7-fba49d709389-tls-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:35.940611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.940561    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5ng\" (UniqueName: \"kubernetes.io/projected/fe34b4ed-614d-4171-a6b7-fba49d709389-kube-api-access-vk5ng\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:35.940611 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:35.940573    4515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe34b4ed-614d-4171-a6b7-fba49d709389-dshm\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:36.094734 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:36.094702    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:54:36.107236 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:36.107201    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-5c7145/simulator-single-main-5c7145-kserve-6648c88b69-fnbhs"]
Nov 02 08:54:37.371189 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:37.371152    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" path="/var/lib/kubelet/pods/fe34b4ed-614d-4171-a6b7-fba49d709389/volumes"
Nov 02 08:54:39.373894 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:39.373856    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:54:39.374299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:39.373901    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:54:46.561118 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.561085    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:54:46.593529 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.593504    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:54:46.647815 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.647781    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\") pod \"a8c8c7fd-d512-47ee-850b-858e6402dd87\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") "
Nov 02 08:54:46.648018 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.647859    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\" (UniqueName: \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\") pod \"a8c8c7fd-d512-47ee-850b-858e6402dd87\" (UID: \"a8c8c7fd-d512-47ee-850b-858e6402dd87\") "
Nov 02 08:54:46.648223 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.648181    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume" pod "a8c8c7fd-d512-47ee-850b-858e6402dd87" (UID: "a8c8c7fd-d512-47ee-850b-858e6402dd87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:54:46.650038 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.650013    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c" pod "a8c8c7fd-d512-47ee-850b-858e6402dd87" (UID: "a8c8c7fd-d512-47ee-850b-858e6402dd87"). InnerVolumeSpecName "kube-api-access-vxh4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:54:46.749276 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.749240    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\" (UniqueName: \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\") pod \"a57e794b-3e9f-4750-a277-3caf231be071\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") "
Nov 02 08:54:46.749480 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.749440    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\") pod \"a57e794b-3e9f-4750-a277-3caf231be071\" (UID: \"a57e794b-3e9f-4750-a277-3caf231be071\") "
Nov 02 08:54:46.749783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.749759    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c8c7fd-d512-47ee-850b-858e6402dd87-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:46.749783 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.749782    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxh4c\" (UniqueName: \"kubernetes.io/projected/a8c8c7fd-d512-47ee-850b-858e6402dd87-kube-api-access-vxh4c\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:46.749947 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.749783    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume" pod "a57e794b-3e9f-4750-a277-3caf231be071" (UID: "a57e794b-3e9f-4750-a277-3caf231be071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:54:46.751428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.751410    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv" pod "a57e794b-3e9f-4750-a277-3caf231be071" (UID: "a57e794b-3e9f-4750-a277-3caf231be071"). InnerVolumeSpecName "kube-api-access-xrzkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:54:46.800934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.800853    4515 generic.go:356] "Generic (PLEG): container finished" podID="a57e794b-3e9f-4750-a277-3caf231be071" containerID="bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b" exitCode=0
Nov 02 08:54:46.800934 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.800898    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"
Nov 02 08:54:46.801138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.800935    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" event={"ID":"a57e794b-3e9f-4750-a277-3caf231be071","Type":"ContainerDied","Data":"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"}
Nov 02 08:54:46.801138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.800988    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9" event={"ID":"a57e794b-3e9f-4750-a277-3caf231be071","Type":"ContainerDied","Data":"a4b9bdac73d27213b21cea9457f4afa390ea3e8f4281dcef20a61ca975b2980d"}
Nov 02 08:54:46.801138 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.801018    4515 scope.go:122] "RemoveContainer" containerID="bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"
Nov 02 08:54:46.802415 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.802389    4515 generic.go:356] "Generic (PLEG): container finished" podID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerID="54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6" exitCode=0
Nov 02 08:54:46.802545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.802415    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" event={"ID":"a8c8c7fd-d512-47ee-850b-858e6402dd87","Type":"ContainerDied","Data":"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"}
Nov 02 08:54:46.802545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.802433    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"
Nov 02 08:54:46.802545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.802453    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf" event={"ID":"a8c8c7fd-d512-47ee-850b-858e6402dd87","Type":"ContainerDied","Data":"d20e5c5fe7ea79171233bd2562a7b81326e44ff5235caf517b075813233c751d"}
Nov 02 08:54:46.809866 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.809843    4515 scope.go:122] "RemoveContainer" containerID="bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"
Nov 02 08:54:46.810128 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:46.810108    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b\": container with ID starting with bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b not found: ID does not exist" containerID="bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"
Nov 02 08:54:46.810199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.810136    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b"} err="failed to get container status \"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b\": rpc error: code = NotFound desc = could not find container \"bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b\": container with ID starting with bde19f5de73049ac7d272c4af5fd0202750290bdf797afe1cbca0c771258087b not found: ID does not exist"
Nov 02 08:54:46.810199 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.810157    4515 scope.go:122] "RemoveContainer" containerID="54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"
Nov 02 08:54:46.818371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.818356    4515 scope.go:122] "RemoveContainer" containerID="54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"
Nov 02 08:54:46.818628 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:46.818613    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6\": container with ID starting with 54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6 not found: ID does not exist" containerID="54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"
Nov 02 08:54:46.818673 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.818634    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6"} err="failed to get container status \"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6\": rpc error: code = NotFound desc = could not find container \"54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6\": container with ID starting with 54eb2e7c8349c8cd6a8ca747ced24f4056fdf6127d0f0b33f56abb9274a400d6 not found: ID does not exist"
Nov 02 08:54:46.850485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.850460    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57e794b-3e9f-4750-a277-3caf231be071-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:46.850485 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.850483    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzkv\" (UniqueName: \"kubernetes.io/projected/a57e794b-3e9f-4750-a277-3caf231be071-kube-api-access-xrzkv\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:54:46.853564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.853541    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:54:46.858254 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.858229    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695j7bp9"]
Nov 02 08:54:46.892723 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.892693    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:54:46.902892 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:46.902867    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdvmpxf"]
Nov 02 08:54:47.372093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:47.372064    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57e794b-3e9f-4750-a277-3caf231be071" path="/var/lib/kubelet/pods/a57e794b-3e9f-4750-a277-3caf231be071/volumes"
Nov 02 08:54:47.372483 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:47.372470    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" path="/var/lib/kubelet/pods/a8c8c7fd-d512-47ee-850b-858e6402dd87/volumes"
Nov 02 08:54:47.722707 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:54:47.722613    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:54:55.665922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.665887    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666262    4515 cpu_manager.go:414] "containerMap: removing container" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666273    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666291    4515 cpu_manager.go:414] "containerMap: removing container" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666296    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666319    4515 cpu_manager.go:414] "containerMap: removing container" podUID="c268428a-2bc8-4b88-94c0-032e828871e1" containerName="maas-api"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666324    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c268428a-2bc8-4b88-94c0-032e828871e1" containerName="maas-api"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666340    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerName="main"
Nov 02 08:54:55.666368 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666345    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerName="main"
Nov 02 08:54:55.666658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666397    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57e794b-3e9f-4750-a277-3caf231be071" containerName="payload-pre-processing"
Nov 02 08:54:55.666658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666406    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c268428a-2bc8-4b88-94c0-032e828871e1" containerName="maas-api"
Nov 02 08:54:55.666658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666414    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe34b4ed-614d-4171-a6b7-fba49d709389" containerName="main"
Nov 02 08:54:55.666658 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.666423    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c8c7fd-d512-47ee-850b-858e6402dd87" containerName="payload-processing"
Nov 02 08:54:55.670863 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.670843    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.690131 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.690099    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5c7145-serving-cert\""
Nov 02 08:54:55.692564 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.692527    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:54:55.733674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.733639    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.733674 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.733677    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.733975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.733703    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd42m\" (UniqueName: \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.733975 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.733820    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.834732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.834696    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.834732 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.834734    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.834959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.834759    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd42m\" (UniqueName: \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.834959 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.834799    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.835544 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.835520    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.837281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.837255    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.837281 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.837273    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.858540 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.858511    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd42m\" (UniqueName: \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\") pod \"maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:55.984922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:55.984843    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:56.219871 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.219834    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:54:56.223580 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.223561    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.239918 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.239847    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.239918 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.239894    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8f9\" (UniqueName: \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.240633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.240610    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-89e08e1e-dockercfg-ktwvz\""
Nov 02 08:54:56.240633 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.240620    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5c7145\""
Nov 02 08:54:56.252301 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.252267    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:54:56.291657 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:54:56.291628    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefaa026_0a81_4122_b6e8_b2a74d180536.slice/crio-e69949cd0bd52384391f008a8dcc9afb7871dba0509305f26f7174f788592af7 WatchSource:0}: Error finding container e69949cd0bd52384391f008a8dcc9afb7871dba0509305f26f7174f788592af7: Status 404 returned error can't find the container with id e69949cd0bd52384391f008a8dcc9afb7871dba0509305f26f7174f788592af7
Nov 02 08:54:56.291799 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.291774    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:54:56.323503 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.323454    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:54:56.330334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.330278    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.344357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.342660    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.344357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.342734    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8f9\" (UniqueName: \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.344357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.342804    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdc6g\" (UniqueName: \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.344357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.342924    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.344357 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.343747    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.350011 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.349982    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:54:56.367017 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.366983    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8f9\" (UniqueName: \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\") pod \"payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.443645 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.443603    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.443841 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.443724    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdc6g\" (UniqueName: \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.444369 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.444351    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.467905 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.467876    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdc6g\" (UniqueName: \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\") pod \"payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.536050 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.536013    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:56.650711 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.650674    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:56.840847 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.840820    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:54:56.841759 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:54:56.841730    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3c121c_3b4d_42c0_bc0a_f007c997fe21.slice/crio-2896d9a0db01d8d3ae4b0fa6073a75985cb13c162618781001e7ed2bc593e233 WatchSource:0}: Error finding container 2896d9a0db01d8d3ae4b0fa6073a75985cb13c162618781001e7ed2bc593e233: Status 404 returned error can't find the container with id 2896d9a0db01d8d3ae4b0fa6073a75985cb13c162618781001e7ed2bc593e233
Nov 02 08:54:56.853533 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.853502    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" event={"ID":"2b3c121c-3b4d-42c0-bc0a-f007c997fe21","Type":"ContainerStarted","Data":"2896d9a0db01d8d3ae4b0fa6073a75985cb13c162618781001e7ed2bc593e233"}
Nov 02 08:54:56.855742 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.855708    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" event={"ID":"fefaa026-0a81-4122-b6e8-b2a74d180536","Type":"ContainerStarted","Data":"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"}
Nov 02 08:54:56.855900 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.855768    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" event={"ID":"fefaa026-0a81-4122-b6e8-b2a74d180536","Type":"ContainerStarted","Data":"e69949cd0bd52384391f008a8dcc9afb7871dba0509305f26f7174f788592af7"}
Nov 02 08:54:56.855992 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.855940    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:54:56.963098 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.963053    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" podStartSLOduration=1.585165632 podStartE2EDuration="1.963041109s" totalImagesPullingTime="377.875477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:54:56.901391034 +0000 UTC m=+5441.663827940" watchObservedRunningTime="2026-11-02 08:54:56.963041109 +0000 UTC m=+5441.725478015"
Nov 02 08:54:56.964823 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:56.964797    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:54:56.964946 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:54:56.964828    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8291903_9409_4e73_ba16_9d92ac75e83b.slice/crio-136e42da52de9a86140cf0db4842289b7f92cac2767b1b0801f6f48c7d14c8d8 WatchSource:0}: Error finding container 136e42da52de9a86140cf0db4842289b7f92cac2767b1b0801f6f48c7d14c8d8: Status 404 returned error can't find the container with id 136e42da52de9a86140cf0db4842289b7f92cac2767b1b0801f6f48c7d14c8d8
Nov 02 08:54:57.862648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.862557    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" event={"ID":"f8291903-9409-4e73-ba16-9d92ac75e83b","Type":"ContainerStarted","Data":"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"}
Nov 02 08:54:57.862648 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.862607    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" event={"ID":"f8291903-9409-4e73-ba16-9d92ac75e83b","Type":"ContainerStarted","Data":"136e42da52de9a86140cf0db4842289b7f92cac2767b1b0801f6f48c7d14c8d8"}
Nov 02 08:54:57.863170 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.862649    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:54:57.864184 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.864154    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" event={"ID":"2b3c121c-3b4d-42c0-bc0a-f007c997fe21","Type":"ContainerStarted","Data":"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"}
Nov 02 08:54:57.864372 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.864238    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:54:57.905067 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.905012    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" podStartSLOduration=1.904994973 podStartE2EDuration="1.904994973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:54:57.903324796 +0000 UTC m=+5442.665761694" watchObservedRunningTime="2026-11-02 08:54:57.904994973 +0000 UTC m=+5442.667431881"
Nov 02 08:54:57.945279 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:57.945228    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" podStartSLOduration=1.9452134540000001 podStartE2EDuration="1.945213454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:54:57.944111812 +0000 UTC m=+5442.706548717" watchObservedRunningTime="2026-11-02 08:54:57.945213454 +0000 UTC m=+5442.707650407"
Nov 02 08:54:59.764979 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:54:59.764937    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:55:02.869827 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:02.869798    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:55:04.725013 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:04.724981    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:55:08.879755 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:08.879724    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:55:08.880286 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:08.879895    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:55:15.018275 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:15.018238    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:55:15.049718 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:15.049683    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:55:16.670803 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.670760    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:55:16.671298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.670975    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" podUID="fefaa026-0a81-4122-b6e8-b2a74d180536" containerName="maas-api" containerID="cri-o://2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b" gracePeriod=30
Nov 02 08:55:16.918566 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.918539    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:55:16.965679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.965601    4515 generic.go:356] "Generic (PLEG): container finished" podID="fefaa026-0a81-4122-b6e8-b2a74d180536" containerID="2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b" exitCode=0
Nov 02 08:55:16.965679 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.965657    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"
Nov 02 08:55:16.965922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.965684    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" event={"ID":"fefaa026-0a81-4122-b6e8-b2a74d180536","Type":"ContainerDied","Data":"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"}
Nov 02 08:55:16.965922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.965741    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r" event={"ID":"fefaa026-0a81-4122-b6e8-b2a74d180536","Type":"ContainerDied","Data":"e69949cd0bd52384391f008a8dcc9afb7871dba0509305f26f7174f788592af7"}
Nov 02 08:55:16.965922 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.965765    4515 scope.go:122] "RemoveContainer" containerID="2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"
Nov 02 08:55:16.976697 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.976665    4515 scope.go:122] "RemoveContainer" containerID="2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"
Nov 02 08:55:16.977057 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:16.977031    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b\": container with ID starting with 2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b not found: ID does not exist" containerID="2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"
Nov 02 08:55:16.977139 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:16.977070    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b"} err="failed to get container status \"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b\": rpc error: code = NotFound desc = could not find container \"2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b\": container with ID starting with 2ad5111ac4b9957a12d56df3aca8ee25a97faee6f7b66ec146da6a17dd44664b not found: ID does not exist"
Nov 02 08:55:17.053052 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053018    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\" (UniqueName: \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\") pod \"fefaa026-0a81-4122-b6e8-b2a74d180536\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") "
Nov 02 08:55:17.053259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053086    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\") pod \"fefaa026-0a81-4122-b6e8-b2a74d180536\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") "
Nov 02 08:55:17.053259 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053124    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\") pod \"fefaa026-0a81-4122-b6e8-b2a74d180536\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") "
Nov 02 08:55:17.053398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053286    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\") pod \"fefaa026-0a81-4122-b6e8-b2a74d180536\" (UID: \"fefaa026-0a81-4122-b6e8-b2a74d180536\") "
Nov 02 08:55:17.053558 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053536    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca" pod "fefaa026-0a81-4122-b6e8-b2a74d180536" (UID: "fefaa026-0a81-4122-b6e8-b2a74d180536"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:55:17.053719 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.053698    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefaa026-0a81-4122-b6e8-b2a74d180536-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:17.055428 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.055401    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m" pod "fefaa026-0a81-4122-b6e8-b2a74d180536" (UID: "fefaa026-0a81-4122-b6e8-b2a74d180536"). InnerVolumeSpecName "kube-api-access-qd42m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:17.055545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.055402    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs" pod "fefaa026-0a81-4122-b6e8-b2a74d180536" (UID: "fefaa026-0a81-4122-b6e8-b2a74d180536"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:55:17.055545 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.055430    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls" pod "fefaa026-0a81-4122-b6e8-b2a74d180536" (UID: "fefaa026-0a81-4122-b6e8-b2a74d180536"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:55:17.155266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.155229    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd42m\" (UniqueName: \"kubernetes.io/projected/fefaa026-0a81-4122-b6e8-b2a74d180536-kube-api-access-qd42m\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:17.155266 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.155264    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:17.155499 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.155276    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefaa026-0a81-4122-b6e8-b2a74d180536-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:17.312701 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.312639    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:55:17.325885 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.325854    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5c7145-5c7d8647c6-rhq9r"]
Nov 02 08:55:17.371794 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.371763    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefaa026-0a81-4122-b6e8-b2a74d180536" path="/var/lib/kubelet/pods/fefaa026-0a81-4122-b6e8-b2a74d180536/volumes"
Nov 02 08:55:17.722873 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:17.722788    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:55:17.899850 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.899816    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:55:17.900094 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:17.900057    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy" containerID="cri-o://f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233" gracePeriod=30
Nov 02 08:55:20.018739 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.018673    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" podUID="f8291903-9409-4e73-ba16-9d92ac75e83b" containerName="payload-processing" containerID="cri-o://e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3" gracePeriod=25
Nov 02 08:55:20.050750 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.050696    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" podUID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" containerName="payload-pre-processing" containerID="cri-o://8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0" gracePeriod=25
Nov 02 08:55:20.093371 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.093338    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:55:20.319285 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.319259    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:55:20.330702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.330678    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:55:20.386490 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.386449    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\" (UniqueName: \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\") pod \"f8291903-9409-4e73-ba16-9d92ac75e83b\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") "
Nov 02 08:55:20.386688 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.386615    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\") pod \"f8291903-9409-4e73-ba16-9d92ac75e83b\" (UID: \"f8291903-9409-4e73-ba16-9d92ac75e83b\") "
Nov 02 08:55:20.387021 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.386998    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume" pod "f8291903-9409-4e73-ba16-9d92ac75e83b" (UID: "f8291903-9409-4e73-ba16-9d92ac75e83b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:55:20.388676 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.388645    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g" pod "f8291903-9409-4e73-ba16-9d92ac75e83b" (UID: "f8291903-9409-4e73-ba16-9d92ac75e83b"). InnerVolumeSpecName "kube-api-access-zdc6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:20.487579 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.487520    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\") pod \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") "
Nov 02 08:55:20.487791 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.487634    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\" (UniqueName: \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\") pod \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\" (UID: \"2b3c121c-3b4d-42c0-bc0a-f007c997fe21\") "
Nov 02 08:55:20.487941 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.487914    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume" pod "2b3c121c-3b4d-42c0-bc0a-f007c997fe21" (UID: "2b3c121c-3b4d-42c0-bc0a-f007c997fe21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:55:20.488009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.487947    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8291903-9409-4e73-ba16-9d92ac75e83b-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:20.488009 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.487967    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdc6g\" (UniqueName: \"kubernetes.io/projected/f8291903-9409-4e73-ba16-9d92ac75e83b-kube-api-access-zdc6g\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:20.489774 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.489752    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9" pod "2b3c121c-3b4d-42c0-bc0a-f007c997fe21" (UID: "2b3c121c-3b4d-42c0-bc0a-f007c997fe21"). InnerVolumeSpecName "kube-api-access-js8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:20.588687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.588605    4515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-plugins-config-volume\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:20.588687 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.588636    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8f9\" (UniqueName: \"kubernetes.io/projected/2b3c121c-3b4d-42c0-bc0a-f007c997fe21-kube-api-access-js8f9\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:20.986117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.986029    4515 generic.go:356] "Generic (PLEG): container finished" podID="f8291903-9409-4e73-ba16-9d92ac75e83b" containerID="e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3" exitCode=0
Nov 02 08:55:20.986117 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.986072    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"
Nov 02 08:55:20.986394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.986112    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" event={"ID":"f8291903-9409-4e73-ba16-9d92ac75e83b","Type":"ContainerDied","Data":"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"}
Nov 02 08:55:20.986394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.986163    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v" event={"ID":"f8291903-9409-4e73-ba16-9d92ac75e83b","Type":"ContainerDied","Data":"136e42da52de9a86140cf0db4842289b7f92cac2767b1b0801f6f48c7d14c8d8"}
Nov 02 08:55:20.986394 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.986189    4515 scope.go:122] "RemoveContainer" containerID="e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"
Nov 02 08:55:20.987553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.987489    4515 generic.go:356] "Generic (PLEG): container finished" podID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" containerID="8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0" exitCode=0
Nov 02 08:55:20.987553 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.987540    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"
Nov 02 08:55:20.987678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.987555    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" event={"ID":"2b3c121c-3b4d-42c0-bc0a-f007c997fe21","Type":"ContainerDied","Data":"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"}
Nov 02 08:55:20.987678 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.987585    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8" event={"ID":"2b3c121c-3b4d-42c0-bc0a-f007c997fe21","Type":"ContainerDied","Data":"2896d9a0db01d8d3ae4b0fa6073a75985cb13c162618781001e7ed2bc593e233"}
Nov 02 08:55:20.994988 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.994965    4515 scope.go:122] "RemoveContainer" containerID="e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"
Nov 02 08:55:20.995271 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:20.995250    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3\": container with ID starting with e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3 not found: ID does not exist" containerID="e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"
Nov 02 08:55:20.995430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.995281    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3"} err="failed to get container status \"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3\": rpc error: code = NotFound desc = could not find container \"e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3\": container with ID starting with e56bc96856392031dae6b429b764914af95977d135e47b4f977353bc32a3fda3 not found: ID does not exist"
Nov 02 08:55:20.995430 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:20.995299    4515 scope.go:122] "RemoveContainer" containerID="8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"
Nov 02 08:55:21.003771 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.003754    4515 scope.go:122] "RemoveContainer" containerID="8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"
Nov 02 08:55:21.004023 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:21.004006    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0\": container with ID starting with 8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0 not found: ID does not exist" containerID="8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"
Nov 02 08:55:21.004087 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.004030    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0"} err="failed to get container status \"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0\": rpc error: code = NotFound desc = could not find container \"8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0\": container with ID starting with 8ed15a634b186ebd292f3726cba60417792ef9767644e230fac1d1b137fda9b0 not found: ID does not exist"
Nov 02 08:55:21.038039 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.038012    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:55:21.050398 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.050370    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5c7145-7bd8885bdkrw5v"]
Nov 02 08:55:21.076514 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.076482    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:55:21.080811 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.080783    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5c7145-5c695qrts8"]
Nov 02 08:55:21.372657 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.372628    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" path="/var/lib/kubelet/pods/2b3c121c-3b4d-42c0-bc0a-f007c997fe21/volumes"
Nov 02 08:55:21.373016 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:21.373003    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8291903-9409-4e73-ba16-9d92ac75e83b" path="/var/lib/kubelet/pods/f8291903-9409-4e73-ba16-9d92ac75e83b/volumes"
Nov 02 08:55:23.159142 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.159110    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:55:23.319093 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319045    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319124    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319162    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319191    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319221    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319249    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319378    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319394    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319412    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319520    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\") pod \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\" (UID: \"cacf1143-0ba7-4f10-a8e2-82bf7b719c48\") "
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319538    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:23.319681 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319640    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:23.319987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319675    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:55:23.319987 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.319871    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:23.320109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.320046    4515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-data\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.320109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.320068    4515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.320109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.320083    4515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-credential-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.320109 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.320098    4515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-workload-socket\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.320272 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.320111    4515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istiod-ca-cert\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.321602 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.321573    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:55:23.321710 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.321606    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:23.321935 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.321921    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "kube-api-access-t5m9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:23.321996 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.321932    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy" pod "cacf1143-0ba7-4f10-a8e2-82bf7b719c48" (UID: "cacf1143-0ba7-4f10-a8e2-82bf7b719c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:23.421505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.421465    4515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-podinfo\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.421505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.421498    4515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-envoy\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.421505 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.421507    4515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-istio-token\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:23.421762 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:23.421515    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5m9d\" (UniqueName: \"kubernetes.io/projected/cacf1143-0ba7-4f10-a8e2-82bf7b719c48-kube-api-access-t5m9d\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:55:24.004112 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.004078    4515 generic.go:356] "Generic (PLEG): container finished" podID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerID="f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233" exitCode=0
Nov 02 08:55:24.004298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.004147    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"
Nov 02 08:55:24.004298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.004151    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerDied","Data":"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"}
Nov 02 08:55:24.004298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.004255    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m" event={"ID":"cacf1143-0ba7-4f10-a8e2-82bf7b719c48","Type":"ContainerDied","Data":"3bd1fe12ca0616825ef4c1051ed74f3fc40efaf45ee7d4d3552be5ada2952baf"}
Nov 02 08:55:24.004298 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.004277    4515 scope.go:122] "RemoveContainer" containerID="f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"
Nov 02 08:55:24.012878 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.012853    4515 scope.go:122] "RemoveContainer" containerID="2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"
Nov 02 08:55:24.020766 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.020544    4515 scope.go:122] "RemoveContainer" containerID="f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"
Nov 02 08:55:24.020857 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:24.020834    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233\": container with ID starting with f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233 not found: ID does not exist" containerID="f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"
Nov 02 08:55:24.020895 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.020865    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233"} err="failed to get container status \"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233\": rpc error: code = NotFound desc = could not find container \"f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233\": container with ID starting with f57229931ca6b681a7d46660c8c1552caf0706ed4e2c54a15270389604b8a233 not found: ID does not exist"
Nov 02 08:55:24.020895 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.020885    4515 scope.go:122] "RemoveContainer" containerID="2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"
Nov 02 08:55:24.021242 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:24.021219    4515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60\": container with ID starting with 2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60 not found: ID does not exist" containerID="2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"
Nov 02 08:55:24.021391 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.021247    4515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60"} err="failed to get container status \"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60\": rpc error: code = NotFound desc = could not find container \"2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60\": container with ID starting with 2b58233aeb7d296a3447383623be7bcac55d466cc15cb29e638b4561d1a7bc60 not found: ID does not exist"
Nov 02 08:55:24.071036 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.070999    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:55:24.083683 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:24.083653    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5c7145-openshift-default-5bbcf8cb9-tpg8m"]
Nov 02 08:55:25.370980 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:25.370947    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" path="/var/lib/kubelet/pods/cacf1143-0ba7-4f10-a8e2-82bf7b719c48/volumes"
Nov 02 08:55:31.446400 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:55:31.446365    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:55:47.722432 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:55:47.722393    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:56:17.722572 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:56:17.722529    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:56:47.722646 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:56:47.722606    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:57:00.242248 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:00.242208    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:57:15.740792 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:15.740749    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:57:17.723071 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:57:17.723034    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:57:21.195340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:21.189826    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:57:25.636086 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:25.636054    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:57:31.425669 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.425638    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"]
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426021    4515 cpu_manager.go:414] "containerMap: removing container" podUID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" containerName="payload-pre-processing"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426032    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" containerName="payload-pre-processing"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426044    4515 cpu_manager.go:414] "containerMap: removing container" podUID="f8291903-9409-4e73-ba16-9d92ac75e83b" containerName="payload-processing"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426050    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8291903-9409-4e73-ba16-9d92ac75e83b" containerName="payload-processing"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426058    4515 cpu_manager.go:414] "containerMap: removing container" podUID="fefaa026-0a81-4122-b6e8-b2a74d180536" containerName="maas-api"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426063    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefaa026-0a81-4122-b6e8-b2a74d180536" containerName="maas-api"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426081    4515 cpu_manager.go:414] "containerMap: removing container" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426086    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426096    4515 cpu_manager.go:414] "containerMap: removing container" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426101    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426166    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3c121c-3b4d-42c0-bc0a-f007c997fe21" containerName="payload-pre-processing"
Nov 02 08:57:31.426168 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426175    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8291903-9409-4e73-ba16-9d92ac75e83b" containerName="payload-processing"
Nov 02 08:57:31.426595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426182    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefaa026-0a81-4122-b6e8-b2a74d180536" containerName="maas-api"
Nov 02 08:57:31.426595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426192    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.426595 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.426199    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacf1143-0ba7-4f10-a8e2-82bf7b719c48" containerName="istio-proxy"
Nov 02 08:57:31.429880 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.429850    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.457691 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.457658    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"]
Nov 02 08:57:31.553702 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.553669    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-maas-api-tls\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.553903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.553712    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6g7\" (UniqueName: \"kubernetes.io/projected/942e1701-53cb-4d5a-a653-3584ee6eabf9-kube-api-access-zs6g7\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.553903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.553733    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942e1701-53cb-4d5a-a653-3584ee6eabf9-openshift-service-ca\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.553903 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.553873    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-metrics-certs\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.655096 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.655063    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-maas-api-tls\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.655299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.655114    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6g7\" (UniqueName: \"kubernetes.io/projected/942e1701-53cb-4d5a-a653-3584ee6eabf9-kube-api-access-zs6g7\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.655299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.655139    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942e1701-53cb-4d5a-a653-3584ee6eabf9-openshift-service-ca\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.655299 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.655175    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-metrics-certs\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.655970 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.655941    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942e1701-53cb-4d5a-a653-3584ee6eabf9-openshift-service-ca\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.657692 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.657666    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-metrics-certs\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.657858 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.657835    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942e1701-53cb-4d5a-a653-3584ee6eabf9-maas-api-tls\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.684181 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.684104    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6g7\" (UniqueName: \"kubernetes.io/projected/942e1701-53cb-4d5a-a653-3584ee6eabf9-kube-api-access-zs6g7\") pod \"maas-api-75cb49fd59-7hhqd\" (UID: \"942e1701-53cb-4d5a-a653-3584ee6eabf9\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:31.746857 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:31.746824    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:32.047624 ip-10-0-137-156 kubenswrapper[4515]: W1102 08:57:32.047595    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942e1701_53cb_4d5a_a653_3584ee6eabf9.slice/crio-eedb31f3ca6b8bf32972b2df55dd091fef8c95c348015e8f0343c351cc69df10 WatchSource:0}: Error finding container eedb31f3ca6b8bf32972b2df55dd091fef8c95c348015e8f0343c351cc69df10: Status 404 returned error can't find the container with id eedb31f3ca6b8bf32972b2df55dd091fef8c95c348015e8f0343c351cc69df10
Nov 02 08:57:32.048175 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:32.048149    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"]
Nov 02 08:57:32.655984 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:32.655945    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd" event={"ID":"942e1701-53cb-4d5a-a653-3584ee6eabf9","Type":"ContainerStarted","Data":"168f7f902b90c5cab1e7c7831cffe87970d66c9aa4d7e8c75963cff6132fb1e8"}
Nov 02 08:57:32.656497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:32.656010    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd" event={"ID":"942e1701-53cb-4d5a-a653-3584ee6eabf9","Type":"ContainerStarted","Data":"eedb31f3ca6b8bf32972b2df55dd091fef8c95c348015e8f0343c351cc69df10"}
Nov 02 08:57:32.656497 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:32.656049    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:32.701819 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:32.701715    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd" podStartSLOduration=1.303627503 podStartE2EDuration="1.701701751s" totalImagesPullingTime="398.074248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:57:32.701291603 +0000 UTC m=+5597.463728511" watchObservedRunningTime="2026-11-02 08:57:32.701701751 +0000 UTC m=+5597.464138657"
Nov 02 08:57:38.667037 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:38.667004    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-7hhqd"
Nov 02 08:57:42.539477 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:42.539436    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:57:46.472496 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:46.472451    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 08:57:46.473054 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:46.472865    4515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" podUID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" containerName="maas-api" containerID="cri-o://c85882f9cf6233caae03f4793a03dcab5a246b37784e70335cfd93744d71789f" gracePeriod=30
Nov 02 08:57:47.747870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:47.747678    4515 generic.go:356] "Generic (PLEG): container finished" podID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" containerID="c85882f9cf6233caae03f4793a03dcab5a246b37784e70335cfd93744d71789f" exitCode=0
Nov 02 08:57:47.747870 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:47.747788    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" event={"ID":"a054e471-aacb-438a-b1c8-f56fdedd3bd2","Type":"ContainerDied","Data":"c85882f9cf6233caae03f4793a03dcab5a246b37784e70335cfd93744d71789f"}
Nov 02 08:57:47.935492 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:57:47.935266    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:57:48.394033 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.393691    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 08:57:48.479629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.478753    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\") pod \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") "
Nov 02 08:57:48.479629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.478821    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\") pod \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") "
Nov 02 08:57:48.479629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.478865    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\") pod \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") "
Nov 02 08:57:48.479629 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.478905    4515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\" (UniqueName: \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\") pod \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\" (UID: \"a054e471-aacb-438a-b1c8-f56fdedd3bd2\") "
Nov 02 08:57:48.495467 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.493239    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca" pod "a054e471-aacb-438a-b1c8-f56fdedd3bd2" (UID: "a054e471-aacb-438a-b1c8-f56fdedd3bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:57:48.516525 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.516484    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs" pod "a054e471-aacb-438a-b1c8-f56fdedd3bd2" (UID: "a054e471-aacb-438a-b1c8-f56fdedd3bd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:57:48.516832 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.516697    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls" pod "a054e471-aacb-438a-b1c8-f56fdedd3bd2" (UID: "a054e471-aacb-438a-b1c8-f56fdedd3bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:57:48.517005 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.516789    4515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj" pod "a054e471-aacb-438a-b1c8-f56fdedd3bd2" (UID: "a054e471-aacb-438a-b1c8-f56fdedd3bd2"). InnerVolumeSpecName "kube-api-access-d7wtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:57:48.588114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.587948    4515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a054e471-aacb-438a-b1c8-f56fdedd3bd2-openshift-service-ca\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:57:48.588114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.587991    4515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-maas-api-tls\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:57:48.588114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.588009    4515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a054e471-aacb-438a-b1c8-f56fdedd3bd2-metrics-certs\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:57:48.588114 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.588025    4515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wtj\" (UniqueName: \"kubernetes.io/projected/a054e471-aacb-438a-b1c8-f56fdedd3bd2-kube-api-access-d7wtj\") on node \"ip-10-0-137-156.ec2.internal\" DevicePath \"\""
Nov 02 08:57:48.765333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.763784    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt" event={"ID":"a054e471-aacb-438a-b1c8-f56fdedd3bd2","Type":"ContainerDied","Data":"472d740d0523ec348f6730b5dc1d765232f0fa6c37b9562e056c9aaf06a186cc"}
Nov 02 08:57:48.765333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.763862    4515 scope.go:122] "RemoveContainer" containerID="c85882f9cf6233caae03f4793a03dcab5a246b37784e70335cfd93744d71789f"
Nov 02 08:57:48.765333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.764050    4515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"
Nov 02 08:57:48.832334 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.830425    4515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 08:57:48.833340 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:48.832807    4515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-k7mqt"]
Nov 02 08:57:49.371805 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:57:49.371770    4515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" path="/var/lib/kubelet/pods/a054e471-aacb-438a-b1c8-f56fdedd3bd2/volumes"
Nov 02 08:58:12.014388 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:58:12.014293    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:58:16.834810 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:58:16.834774    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:58:17.735696 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:58:17.735649    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:58:33.944362 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:58:33.944203    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:58:38.718229 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:58:38.718193    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:58:45.439333 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:58:45.438389    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:58:47.721863 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:58:47.721823    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:59:04.821176 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:59:04.821136    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:59:15.131830 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:59:15.131789    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:59:17.723148 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:59:17.723107    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:59:29.733432 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:59:29.733346    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:59:38.136773 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:59:38.136734    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 08:59:47.721353 ip-10-0-137-156 kubenswrapper[4515]: E1102 08:59:47.721297    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 08:59:56.439560 ip-10-0-137-156 kubenswrapper[4515]: I1102 08:59:56.439524    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:00:06.742971 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:00:06.742930    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:00:12.522109 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:00:12.522071    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:00:17.723612 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:00:17.723571    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:00:25.418543 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:00:25.418505    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:00:43.834540 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:00:43.834505    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:00:47.722679 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:00:47.722640    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:01:17.722294 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:01:17.722210    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:01:27.516338 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.514439    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jl6c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:01:27.516338 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.514553    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jl6c9" podUID="3629c3ef-7d97-4239-a095-71741d6885af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:01:27.553723 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.553681    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-66v28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:01:27.554038 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.554002    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-66v28" podUID="650daaeb-7c93-4844-8659-6bebaa064303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:01:27.601437 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.601397    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wgbb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:01:27.602105 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.601705    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wgbb7" podUID="a63a70fa-8feb-47e4-8cef-fd3b1b234b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:01:27.765900 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.765420    4515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-htjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:01:27.765900 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:01:27.765491    4515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-htjfk" podUID="96f9d878-fb9d-437c-b81d-defc288cc2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:01:47.828343 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:01:47.827490    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:02:17.737489 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:02:17.737428    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:02:37.334873 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:02:37.334781    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d5hvn"]
Nov 02 09:02:47.724274 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:02:47.724237    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:03:00.377173 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:00.377138    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-785d49bf89-46p4s_dc11f2c7-aac8-4213-aa09-42b98173d446/manager/0.log"
Nov 02 09:03:01.283882 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:01.283842    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f6b5c786d-z74vw_21eed327-37c2-4e36-8964-d9618c41cb58/manager/0.log"
Nov 02 09:03:01.914934 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:01.914899    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5gs_cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df/proxy/0.log"
Nov 02 09:03:03.058925 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:03.058893    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75cb49fd59-7hhqd_942e1701-53cb-4d5a-a653-3584ee6eabf9/maas-api/0.log"
Nov 02 09:03:09.522176 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:09.522143    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d5hvn_33c78a7e-3e83-4a53-ab31-75f0ef12ade5/limitador/0.log"
Nov 02 09:03:11.224639 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:11.224597    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-sjbdk_067d8042-97a7-4a81-83d5-06414565ddca/kube-auth-proxy/0.log"
Nov 02 09:03:11.406560 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:11.406519    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-66v28_650daaeb-7c93-4844-8659-6bebaa064303/istio-proxy/0.log"
Nov 02 09:03:12.169787 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:12.169752    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-htjfk_96f9d878-fb9d-437c-b81d-defc288cc2be/istio-proxy/0.log"
Nov 02 09:03:12.383186 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:12.383152    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jl6c9_3629c3ef-7d97-4239-a095-71741d6885af/istio-proxy/0.log"
Nov 02 09:03:13.216990 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:13.216955    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wgbb7_a63a70fa-8feb-47e4-8cef-fd3b1b234b5a/istio-proxy/0.log"
Nov 02 09:03:13.589668 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:13.589615    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6fcc67b9b4-qld9t_9c260812-e09d-47bd-83b0-1dd3b85b5058/payload-pre-processing/0.log"
Nov 02 09:03:13.764283 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:13.764250    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d9486d65f-f62bl_0da19d73-3cbd-4001-9425-b502bfb43524/payload-processing/0.log"
Nov 02 09:03:15.070735 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:15.070698    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-c56z2_d2c2ca8c-58ad-4e92-9d4b-caab320c14d5/main/0.log"
Nov 02 09:03:15.640858 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:15.640827    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-glvhv_8e7c6a8a-a628-4320-9f42-726b3469c007/main/0.log"
Nov 02 09:03:15.837542 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:15.837500    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84f2m6_0724f5a8-427f-474a-8f84-74582d627536/main/0.log"
Nov 02 09:03:16.017051 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:16.017012    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-xbgtq_047a025e-3d5b-468c-ae1c-b11786ec507c/main/0.log"
Nov 02 09:03:17.722795 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:03:17.722757    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:03:26.309899 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.309858    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ns4zm/must-gather-zcw8b"]
Nov 02 09:03:26.310385 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.310256    4515 cpu_manager.go:414] "containerMap: removing container" podUID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" containerName="maas-api"
Nov 02 09:03:26.310385 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.310267    4515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" containerName="maas-api"
Nov 02 09:03:26.310385 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.310367    4515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a054e471-aacb-438a-b1c8-f56fdedd3bd2" containerName="maas-api"
Nov 02 09:03:26.313671 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.313650    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25t77/must-gather-fk74g"]
Nov 02 09:03:26.313791 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.313774    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.317247 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.317223    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.332113 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.332089    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ns4zm\"/\"default-dockercfg-5xb2n\""
Nov 02 09:03:26.340340 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.335049    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ns4zm/must-gather-zcw8b"]
Nov 02 09:03:26.344660 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.344634    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ns4zm\"/\"openshift-service-ca.crt\""
Nov 02 09:03:26.344660 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.344650    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ns4zm\"/\"kube-root-ca.crt\""
Nov 02 09:03:26.344949 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.344819    4515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25t77\"/\"default-dockercfg-tsbd8\""
Nov 02 09:03:26.346459 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.346436    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/must-gather-fk74g"]
Nov 02 09:03:26.366843 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.366805    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtkx\" (UniqueName: \"kubernetes.io/projected/b8fbb854-6665-470d-b23b-32f5ad72b748-kube-api-access-hdtkx\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.366843 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.366844    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-must-gather-output\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.367110 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.366870    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmv8\" (UniqueName: \"kubernetes.io/projected/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-kube-api-access-sjmv8\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.367110 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.367082    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8fbb854-6665-470d-b23b-32f5ad72b748-must-gather-output\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.389835 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.389802    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"kube-root-ca.crt\""
Nov 02 09:03:26.390002 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.389895    4515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"openshift-service-ca.crt\""
Nov 02 09:03:26.467714 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.467660    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8fbb854-6665-470d-b23b-32f5ad72b748-must-gather-output\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.467714 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.467729    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtkx\" (UniqueName: \"kubernetes.io/projected/b8fbb854-6665-470d-b23b-32f5ad72b748-kube-api-access-hdtkx\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.468011 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.467758    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-must-gather-output\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.468011 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.467785    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmv8\" (UniqueName: \"kubernetes.io/projected/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-kube-api-access-sjmv8\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.468133 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.468058    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8fbb854-6665-470d-b23b-32f5ad72b748-must-gather-output\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.468176 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.468131    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-must-gather-output\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.493744 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.493715    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtkx\" (UniqueName: \"kubernetes.io/projected/b8fbb854-6665-470d-b23b-32f5ad72b748-kube-api-access-hdtkx\") pod \"must-gather-fk74g\" (UID: \"b8fbb854-6665-470d-b23b-32f5ad72b748\") " pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.493879 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.493764    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmv8\" (UniqueName: \"kubernetes.io/projected/2dce1092-312b-4a11-9af2-f1a8fdc6e0d9-kube-api-access-sjmv8\") pod \"must-gather-zcw8b\" (UID: \"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9\") " pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.639499 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.639406    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ns4zm/must-gather-zcw8b"
Nov 02 09:03:26.648500 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.648480    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/must-gather-fk74g"
Nov 02 09:03:26.989524 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.989494    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ns4zm/must-gather-zcw8b"]
Nov 02 09:03:26.990937 ip-10-0-137-156 kubenswrapper[4515]: W1102 09:03:26.990909    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dce1092_312b_4a11_9af2_f1a8fdc6e0d9.slice/crio-8285bd1695f23178ca6b784541a8b7be920d8ed4a9b40f877e35036daf82f863 WatchSource:0}: Error finding container 8285bd1695f23178ca6b784541a8b7be920d8ed4a9b40f877e35036daf82f863: Status 404 returned error can't find the container with id 8285bd1695f23178ca6b784541a8b7be920d8ed4a9b40f877e35036daf82f863
Nov 02 09:03:26.994097 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.994069    4515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:03:26.994817 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:26.994789    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/must-gather-fk74g"]
Nov 02 09:03:26.995594 ip-10-0-137-156 kubenswrapper[4515]: W1102 09:03:26.995564    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8fbb854_6665_470d_b23b_32f5ad72b748.slice/crio-07c6d1511e5441c950e2af799d966cdc4baecef61fcd39499421186bf5f00fa7 WatchSource:0}: Error finding container 07c6d1511e5441c950e2af799d966cdc4baecef61fcd39499421186bf5f00fa7: Status 404 returned error can't find the container with id 07c6d1511e5441c950e2af799d966cdc4baecef61fcd39499421186bf5f00fa7
Nov 02 09:03:27.624063 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:27.624023    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ns4zm/must-gather-zcw8b" event={"ID":"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9","Type":"ContainerStarted","Data":"8285bd1695f23178ca6b784541a8b7be920d8ed4a9b40f877e35036daf82f863"}
Nov 02 09:03:27.625380 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:27.625350    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/must-gather-fk74g" event={"ID":"b8fbb854-6665-470d-b23b-32f5ad72b748","Type":"ContainerStarted","Data":"07c6d1511e5441c950e2af799d966cdc4baecef61fcd39499421186bf5f00fa7"}
Nov 02 09:03:28.631698 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:28.631659    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/must-gather-fk74g" event={"ID":"b8fbb854-6665-470d-b23b-32f5ad72b748","Type":"ContainerStarted","Data":"ceafd27bbb066c7a5477cad74559492d030d75ebf918d22744e46ee69f9b62f1"}
Nov 02 09:03:28.631698 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:28.631702    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/must-gather-fk74g" event={"ID":"b8fbb854-6665-470d-b23b-32f5ad72b748","Type":"ContainerStarted","Data":"6dfb33e4489bdfb7527d0a80e9cb182cb12f233b59ca3aee136666d4544ad7d3"}
Nov 02 09:03:28.673501 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:28.673438    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-25t77/must-gather-fk74g" podStartSLOduration=1.909490175 podStartE2EDuration="2.673419495s" totalImagesPullingTime="763.92932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:03:28.672931846 +0000 UTC m=+5953.435368811" watchObservedRunningTime="2026-11-02 09:03:28.673419495 +0000 UTC m=+5953.435856403"
Nov 02 09:03:30.953629 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:30.953594    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-brmn4_a3daa4be-adf6-4e9f-8307-98e3c0d180ec/global-pull-secret-syncer/0.log"
Nov 02 09:03:31.996343 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:31.995688    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qfghm_0f836f8b-5b38-4697-b577-46cc16c02f12/konnectivity-agent/0.log"
Nov 02 09:03:32.363332 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:32.362974    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-156.ec2.internal_735495427ccc6de5684bdc4a015d248e/haproxy/0.log"
Nov 02 09:03:40.957790 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:40.957342    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d5hvn_33c78a7e-3e83-4a53-ab31-75f0ef12ade5/limitador/0.log"
Nov 02 09:03:42.766009 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.763993    4515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"]
Nov 02 09:03:42.778770 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.778067    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.787145 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.786963    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"]
Nov 02 09:03:42.818218 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.818112    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-podres\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.818460 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.818240    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-sys\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.818460 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.818287    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhbc\" (UniqueName: \"kubernetes.io/projected/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-kube-api-access-8jhbc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.818460 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.818372    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-proc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.818460 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.818426    4515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-lib-modules\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.924028 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.923883    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-podres\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.929721 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.924006    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-podres\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.929977 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.929750    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-sys\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930126 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.930018    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhbc\" (UniqueName: \"kubernetes.io/projected/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-kube-api-access-8jhbc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930126 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.930087    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-proc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930249 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.930133    4515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-lib-modules\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930350 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.930332    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-lib-modules\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930537 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.930521    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-proc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.930627 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.929825    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-sys\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:42.954707 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:42.954659    4515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhbc\" (UniqueName: \"kubernetes.io/projected/fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe-kube-api-access-8jhbc\") pod \"perf-node-gather-daemonset-6jtx2\" (UID: \"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:43.142717 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:43.142625    4515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:43.458992 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:43.458959    4515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"]
Nov 02 09:03:43.459525 ip-10-0-137-156 kubenswrapper[4515]: W1102 09:03:43.459493    4515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa6cc4e3_51b8_4eac_8fc2_34e8033c64fe.slice/crio-926c1c96e1381130ddcffd39bc1f83e68758b12a55bfa24414a9c7ce8eebd93c WatchSource:0}: Error finding container 926c1c96e1381130ddcffd39bc1f83e68758b12a55bfa24414a9c7ce8eebd93c: Status 404 returned error can't find the container with id 926c1c96e1381130ddcffd39bc1f83e68758b12a55bfa24414a9c7ce8eebd93c
Nov 02 09:03:43.981248 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:43.981207    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2" event={"ID":"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe","Type":"ContainerStarted","Data":"291f65c1bd9e3a97436dce5b1a0c071d6940fa77f9de2f6c330e72ef438db430"}
Nov 02 09:03:43.981668 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:43.981275    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2" event={"ID":"fa6cc4e3-51b8-4eac-8fc2-34e8033c64fe","Type":"ContainerStarted","Data":"926c1c96e1381130ddcffd39bc1f83e68758b12a55bfa24414a9c7ce8eebd93c"}
Nov 02 09:03:43.981668 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:43.981329    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:44.023932 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:44.023865    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2" podStartSLOduration=2.023847726 podStartE2EDuration="2.023847726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:03:44.021940709 +0000 UTC m=+5968.784377616" watchObservedRunningTime="2026-11-02 09:03:44.023847726 +0000 UTC m=+5968.786284632"
Nov 02 09:03:44.991857 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:44.991793    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ns4zm/must-gather-zcw8b" event={"ID":"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9","Type":"ContainerStarted","Data":"83ba96b04711cbf99513dc6ee8226de2b070f8666c3de24c7a2a3d4a726b3071"}
Nov 02 09:03:45.182232 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:45.182158    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs52m_e5d75c53-9177-4bbb-a6f8-4f5d358083c2/node-exporter/0.log"
Nov 02 09:03:45.307615 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:45.307585    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs52m_e5d75c53-9177-4bbb-a6f8-4f5d358083c2/kube-rbac-proxy/0.log"
Nov 02 09:03:45.434801 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:45.434722    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rs52m_e5d75c53-9177-4bbb-a6f8-4f5d358083c2/init-textfile/0.log"
Nov 02 09:03:45.997925 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:45.997881    4515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ns4zm/must-gather-zcw8b" event={"ID":"2dce1092-312b-4a11-9af2-f1a8fdc6e0d9","Type":"ContainerStarted","Data":"99cdeea85ac69d28783c716d0877f04a36ab59b7be236b5921fdfb5099aa8992"}
Nov 02 09:03:46.040213 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:46.040153    4515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ns4zm/must-gather-zcw8b" podStartSLOduration=2.26052343 podStartE2EDuration="20.04013919s" totalImagesPullingTime="17.77961576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:03:46.039070481 +0000 UTC m=+5970.801507388" watchObservedRunningTime="2026-11-02 09:03:46.04013919 +0000 UTC m=+5970.802576095"
Nov 02 09:03:46.883973 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:46.883939    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wszcp_296b6a48-acbb-4fc0-b41e-f03d344e8505/prometheus-operator/0.log"
Nov 02 09:03:46.996067 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:46.996020    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wszcp_296b6a48-acbb-4fc0-b41e-f03d344e8505/kube-rbac-proxy/0.log"
Nov 02 09:03:47.091344 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:47.091295    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-785d49bf89-46p4s_dc11f2c7-aac8-4213-aa09-42b98173d446/manager/0.log"
Nov 02 09:03:47.141333 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:47.141232    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-vnmcv_d0673a2b-eb6a-4d22-af35-c12eaf913948/prometheus-operator-admission-webhook/0.log"
Nov 02 09:03:47.725144 ip-10-0-137-156 kubenswrapper[4515]: E1102 09:03:47.725103    4515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc2t9" containerName="extractor"
Nov 02 09:03:48.183576 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:48.183536    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f6b5c786d-z74vw_21eed327-37c2-4e36-8964-d9618c41cb58/manager/0.log"
Nov 02 09:03:48.414496 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:48.414460    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5gs_cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df/proxy/0.log"
Nov 02 09:03:50.003559 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:50.003530    4515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-6jtx2"
Nov 02 09:03:50.191210 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:50.190359    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-785d49bf89-46p4s_dc11f2c7-aac8-4213-aa09-42b98173d446/manager/0.log"
Nov 02 09:03:51.290920 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:51.290884    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f6b5c786d-z74vw_21eed327-37c2-4e36-8964-d9618c41cb58/manager/0.log"
Nov 02 09:03:51.506585 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:51.506554    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5gs_cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df/proxy/0.log"
Nov 02 09:03:52.350502 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:52.350464    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f5474bc87-l8b5v_8d9b444e-b160-4070-8860-850170ffb81a/console/0.log"
Nov 02 09:03:52.509675 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:52.509552    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-l9ngq_02f3384d-e450-4445-929f-ea4b686859d9/download-server/0.log"
Nov 02 09:03:53.327333 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:53.326570    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-785d49bf89-46p4s_dc11f2c7-aac8-4213-aa09-42b98173d446/manager/0.log"
Nov 02 09:03:54.439411 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:54.439377    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f6b5c786d-z74vw_21eed327-37c2-4e36-8964-d9618c41cb58/manager/0.log"
Nov 02 09:03:54.684441 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:54.684395    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5gs_cdf2fa24-d0ad-4baa-a3ef-b883fa6ca9df/proxy/0.log"
Nov 02 09:03:56.596597 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:56.596552    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rzs7_88cae88a-5ad3-4932-b27e-1a012e018d55/dns/0.log"
Nov 02 09:03:56.707573 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:56.707537    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rzs7_88cae88a-5ad3-4932-b27e-1a012e018d55/kube-rbac-proxy/0.log"
Nov 02 09:03:57.333249 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.333193    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-sjbdk_067d8042-97a7-4a81-83d5-06414565ddca/kube-auth-proxy/0.log"
Nov 02 09:03:57.381014 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.380981    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97v78_71b02e54-cab8-41d7-909c-7f0cbe5eab71/dns-node-resolver/0.log"
Nov 02 09:03:57.409108 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.409070    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-66v28_650daaeb-7c93-4844-8659-6bebaa064303/istio-proxy/0.log"
Nov 02 09:03:57.671235 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.671137    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-htjfk_96f9d878-fb9d-437c-b81d-defc288cc2be/istio-proxy/0.log"
Nov 02 09:03:57.706891 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.706857    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-sjbdk_067d8042-97a7-4a81-83d5-06414565ddca/kube-auth-proxy/0.log"
Nov 02 09:03:57.740985 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.740937    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jl6c9_3629c3ef-7d97-4239-a095-71741d6885af/istio-proxy/0.log"
Nov 02 09:03:57.792354 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.792303    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-66v28_650daaeb-7c93-4844-8659-6bebaa064303/istio-proxy/0.log"
Nov 02 09:03:57.991052 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:57.990958    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wgbb7_a63a70fa-8feb-47e4-8cef-fd3b1b234b5a/istio-proxy/0.log"
Nov 02 09:03:58.055108 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.055079    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-htjfk_96f9d878-fb9d-437c-b81d-defc288cc2be/istio-proxy/0.log"
Nov 02 09:03:58.124205 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.124166    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jl6c9_3629c3ef-7d97-4239-a095-71741d6885af/istio-proxy/0.log"
Nov 02 09:03:58.200145 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.200097    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6fcc67b9b4-qld9t_9c260812-e09d-47bd-83b0-1dd3b85b5058/payload-pre-processing/0.log"
Nov 02 09:03:58.261279 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.261227    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d9486d65f-f62bl_0da19d73-3cbd-4001-9425-b502bfb43524/payload-processing/0.log"
Nov 02 09:03:58.378031 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.376435    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wgbb7_a63a70fa-8feb-47e4-8cef-fd3b1b234b5a/istio-proxy/0.log"
Nov 02 09:03:58.508136 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.508095    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6fcc67b9b4-qld9t_9c260812-e09d-47bd-83b0-1dd3b85b5058/payload-pre-processing/0.log"
Nov 02 09:03:58.586057 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.585658    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d9486d65f-f62bl_0da19d73-3cbd-4001-9425-b502bfb43524/payload-processing/0.log"
Nov 02 09:03:58.831516 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:58.831474    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56f58c559d-klv5k_e4ab6de3-44f6-455d-8345-8eda5144cdce/registry/0.log"
Nov 02 09:03:59.181636 ip-10-0-137-156 kubenswrapper[4515]: I1102 09:03:59.181601    4515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mldbl_903d010c-458d-4c3a-8c94-8bf685cb6ef3/node-ca/0.log"